[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250116 02:57:08.441074 19386 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.238.190:33651
I20250116 02:57:08.442658 19386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:57:08.443730 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:57:08.461325 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:08.462343 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:08.461454 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:08.462749 19386 server_base.cc:1034] running on GCE node
I20250116 02:57:09.594440 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:57:09.594864 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:57:09.595073 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996229595045 us; error 0 us; skew 500 ppm
I20250116 02:57:09.596086 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:57:09.609031 19386 webserver.cc:458] Webserver started at http://127.18.238.190:36021/ using document root <none> and password file <none>
I20250116 02:57:09.610069 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:57:09.610313 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:57:09.610829 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:57:09.618247 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/master-0-root/instance:
uuid: "534a60b857234a638acfeeff7be83bda"
format_stamp: "Formatted at 2025-01-16 02:57:09 on dist-test-slave-qn1x"
I20250116 02:57:09.627803 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20250116 02:57:09.634034 19401 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:09.635707 19386 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250116 02:57:09.636152 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/master-0-root
uuid: "534a60b857234a638acfeeff7be83bda"
format_stamp: "Formatted at 2025-01-16 02:57:09 on dist-test-slave-qn1x"
I20250116 02:57:09.636554 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:57:09.697572 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:57:09.699510 19386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:57:09.700135 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:57:09.783141 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.190:33651
I20250116 02:57:09.783252 19452 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.190:33651 every 8 connection(s)
I20250116 02:57:09.790501 19453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:09.818848 19453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda: Bootstrap starting.
I20250116 02:57:09.825830 19453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:09.828055 19453 log.cc:826] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda: Log is configured to *not* fsync() on all Append() calls
I20250116 02:57:09.833689 19453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda: No bootstrap required, opened a new log
I20250116 02:57:09.859056 19453 raft_consensus.cc:357] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534a60b857234a638acfeeff7be83bda" member_type: VOTER }
I20250116 02:57:09.859757 19453 raft_consensus.cc:383] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:09.859969 19453 raft_consensus.cc:738] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534a60b857234a638acfeeff7be83bda, State: Initialized, Role: FOLLOWER
I20250116 02:57:09.860850 19453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534a60b857234a638acfeeff7be83bda" member_type: VOTER }
I20250116 02:57:09.861404 19453 raft_consensus.cc:397] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250116 02:57:09.861680 19453 raft_consensus.cc:491] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250116 02:57:09.862035 19453 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:09.866739 19453 raft_consensus.cc:513] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534a60b857234a638acfeeff7be83bda" member_type: VOTER }
I20250116 02:57:09.867688 19453 leader_election.cc:304] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 534a60b857234a638acfeeff7be83bda; no voters:
I20250116 02:57:09.869742 19453 leader_election.cc:290] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [CANDIDATE]: Term 1 election: Requested vote from peers
I20250116 02:57:09.870216 19456 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:09.872880 19456 raft_consensus.cc:695] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 1 LEADER]: Becoming Leader. State: Replica: 534a60b857234a638acfeeff7be83bda, State: Running, Role: LEADER
I20250116 02:57:09.873999 19456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534a60b857234a638acfeeff7be83bda" member_type: VOTER }
I20250116 02:57:09.875499 19453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [sys.catalog]: configured and running, proceeding with master startup.
I20250116 02:57:09.885252 19458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 534a60b857234a638acfeeff7be83bda. Latest consensus state: current_term: 1 leader_uuid: "534a60b857234a638acfeeff7be83bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534a60b857234a638acfeeff7be83bda" member_type: VOTER } }
I20250116 02:57:09.886322 19458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [sys.catalog]: This master's current role is: LEADER
I20250116 02:57:09.889348 19457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "534a60b857234a638acfeeff7be83bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534a60b857234a638acfeeff7be83bda" member_type: VOTER } }
I20250116 02:57:09.890159 19457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [sys.catalog]: This master's current role is: LEADER
I20250116 02:57:09.893457 19464 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250116 02:57:09.907371 19464 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250116 02:57:09.909462 19386 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250116 02:57:09.927078 19464 catalog_manager.cc:1348] Generated new cluster ID: 3769e6507ff64e228df3f0534bc6c4d0
I20250116 02:57:09.927455 19464 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250116 02:57:09.964728 19464 catalog_manager.cc:1371] Generated new certificate authority record
I20250116 02:57:09.967308 19464 catalog_manager.cc:1505] Loading token signing keys...
I20250116 02:57:09.989322 19464 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda: Generated new TSK 0
I20250116 02:57:09.990509 19464 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250116 02:57:10.042106 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:57:10.053364 19475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:10.056573 19476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:10.058744 19386 server_base.cc:1034] running on GCE node
W20250116 02:57:10.059193 19478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:10.060423 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:57:10.060786 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:57:10.061010 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996230060987 us; error 0 us; skew 500 ppm
I20250116 02:57:10.061911 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:57:10.067204 19386 webserver.cc:458] Webserver started at http://127.18.238.129:34045/ using document root <none> and password file <none>
I20250116 02:57:10.068048 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:57:10.068339 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:57:10.068720 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:57:10.070464 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-0-root/instance:
uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
format_stamp: "Formatted at 2025-01-16 02:57:10 on dist-test-slave-qn1x"
I20250116 02:57:10.078850 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250116 02:57:10.084718 19483 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:10.085574 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20250116 02:57:10.085884 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-0-root
uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
format_stamp: "Formatted at 2025-01-16 02:57:10 on dist-test-slave-qn1x"
I20250116 02:57:10.086192 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:57:10.112385 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:57:10.113989 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:57:10.116803 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:57:10.121903 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:57:10.122190 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:10.122464 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:57:10.122681 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250116 02:57:10.228814 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.129:38935
I20250116 02:57:10.228924 19545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.129:38935 every 8 connection(s)
I20250116 02:57:10.237068 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:57:10.249218 19550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:10.253760 19551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:10.260538 19553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:10.262513 19386 server_base.cc:1034] running on GCE node
I20250116 02:57:10.263727 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:57:10.264019 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:57:10.264228 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996230264206 us; error 0 us; skew 500 ppm
I20250116 02:57:10.265077 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:57:10.271279 19386 webserver.cc:458] Webserver started at http://127.18.238.130:44703/ using document root <none> and password file <none>
I20250116 02:57:10.272102 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:57:10.272392 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:57:10.273022 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:57:10.274513 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-1-root/instance:
uuid: "d5143dfed70340c3a0453c08ffd663c7"
format_stamp: "Formatted at 2025-01-16 02:57:10 on dist-test-slave-qn1x"
I20250116 02:57:10.278374 19546 heartbeater.cc:346] Connected to a master server at 127.18.238.190:33651
I20250116 02:57:10.278973 19546 heartbeater.cc:463] Registering TS with master...
I20250116 02:57:10.280362 19546 heartbeater.cc:509] Master 127.18.238.190:33651 requested a full tablet report, sending...
I20250116 02:57:10.282809 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250116 02:57:10.284623 19418 ts_manager.cc:194] Registered new tserver with Master: f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935)
I20250116 02:57:10.287784 19558 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:10.288126 19418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56200
I20250116 02:57:10.288759 19386 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250116 02:57:10.289076 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-1-root
uuid: "d5143dfed70340c3a0453c08ffd663c7"
format_stamp: "Formatted at 2025-01-16 02:57:10 on dist-test-slave-qn1x"
I20250116 02:57:10.289395 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:57:10.315398 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:57:10.316916 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:57:10.319710 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:57:10.323129 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:57:10.323359 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:10.323630 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:57:10.323822 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:10.428390 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.130:33723
I20250116 02:57:10.429018 19620 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.130:33723 every 8 connection(s)
I20250116 02:57:10.436131 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:57:10.446194 19624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:10.452425 19621 heartbeater.cc:346] Connected to a master server at 127.18.238.190:33651
I20250116 02:57:10.453297 19386 server_base.cc:1034] running on GCE node
I20250116 02:57:10.453661 19621 heartbeater.cc:463] Registering TS with master...
W20250116 02:57:10.453262 19625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:10.454793 19621 heartbeater.cc:509] Master 127.18.238.190:33651 requested a full tablet report, sending...
W20250116 02:57:10.455359 19627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:10.456542 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:57:10.456918 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:57:10.457135 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996230457113 us; error 0 us; skew 500 ppm
I20250116 02:57:10.457615 19418 ts_manager.cc:194] Registered new tserver with Master: d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:10.457952 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:57:10.459535 19418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56210
I20250116 02:57:10.461871 19386 webserver.cc:458] Webserver started at http://127.18.238.131:36729/ using document root <none> and password file <none>
I20250116 02:57:10.462785 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:57:10.463035 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:57:10.463362 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:57:10.465119 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-2-root/instance:
uuid: "0c459e7a7a4648eca82c6d239d896df6"
format_stamp: "Formatted at 2025-01-16 02:57:10 on dist-test-slave-qn1x"
I20250116 02:57:10.472317 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250116 02:57:10.477908 19632 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:10.479038 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250116 02:57:10.479444 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-2-root
uuid: "0c459e7a7a4648eca82c6d239d896df6"
format_stamp: "Formatted at 2025-01-16 02:57:10 on dist-test-slave-qn1x"
I20250116 02:57:10.479887 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736996228253955-19386-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:57:10.533466 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:57:10.535046 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:57:10.536733 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:57:10.539587 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:57:10.539819 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:10.540071 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:57:10.540241 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:10.659092 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.131:39839
I20250116 02:57:10.659174 19694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.131:39839 every 8 connection(s)
I20250116 02:57:10.679746 19695 heartbeater.cc:346] Connected to a master server at 127.18.238.190:33651
I20250116 02:57:10.680141 19695 heartbeater.cc:463] Registering TS with master...
I20250116 02:57:10.680996 19695 heartbeater.cc:509] Master 127.18.238.190:33651 requested a full tablet report, sending...
I20250116 02:57:10.683128 19418 ts_manager.cc:194] Registered new tserver with Master: 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:10.683954 19386 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017297564s
I20250116 02:57:10.684770 19418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56218
I20250116 02:57:10.688709 19386 test_util.cc:274] Using random seed: -212963423
I20250116 02:57:10.746475 19418 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56226:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250116 02:57:10.750862 19418 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250116 02:57:10.885932 19586 tablet_service.cc:1467] Processing CreateTablet for tablet ffc97072515448ee8d301233977e6086 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250116 02:57:10.887169 19659 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba892d976584027b263e24ff2303c74 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250116 02:57:10.886076 19660 tablet_service.cc:1467] Processing CreateTablet for tablet ffc97072515448ee8d301233977e6086 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250116 02:57:10.887701 19583 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8cbd4766d94af88d1a276ec7f1ecc1 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250116 02:57:10.888871 19660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc97072515448ee8d301233977e6086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.889214 19582 tablet_service.cc:1467] Processing CreateTablet for tablet c722958228fb4aec8e0315ff6fc3595f (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250116 02:57:10.902454 19586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc97072515448ee8d301233977e6086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.904125 19659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ba892d976584027b263e24ff2303c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.906116 19655 tablet_service.cc:1467] Processing CreateTablet for tablet 46904c6b96944fffa7561238601af1d9 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250116 02:57:10.916477 19655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46904c6b96944fffa7561238601af1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.918051 19581 tablet_service.cc:1467] Processing CreateTablet for tablet 46904c6b96944fffa7561238601af1d9 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250116 02:57:10.922159 19579 tablet_service.cc:1467] Processing CreateTablet for tablet e44eff29754e42feaafb330b0cfa2fac (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250116 02:57:10.912997 19585 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba892d976584027b263e24ff2303c74 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250116 02:57:10.913831 19658 tablet_service.cc:1467] Processing CreateTablet for tablet d37284a6220e4e3198e6f6e149a22233 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250116 02:57:10.924546 19582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c722958228fb4aec8e0315ff6fc3595f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.934029 19583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cbd4766d94af88d1a276ec7f1ecc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.918097 19584 tablet_service.cc:1467] Processing CreateTablet for tablet d37284a6220e4e3198e6f6e149a22233 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250116 02:57:10.931070 19509 tablet_service.cc:1467] Processing CreateTablet for tablet d37284a6220e4e3198e6f6e149a22233 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250116 02:57:10.922211 19652 tablet_service.cc:1467] Processing CreateTablet for tablet 5604a103d76a4e8187cd388a9c94f99c (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250116 02:57:10.929930 19656 tablet_service.cc:1467] Processing CreateTablet for tablet c722958228fb4aec8e0315ff6fc3595f (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250116 02:57:10.936278 19653 tablet_service.cc:1467] Processing CreateTablet for tablet e44eff29754e42feaafb330b0cfa2fac (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250116 02:57:10.935014 19510 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba892d976584027b263e24ff2303c74 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250116 02:57:10.929620 19580 tablet_service.cc:1467] Processing CreateTablet for tablet 10d32b1f88a2427d989bc38631930ad2 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250116 02:57:10.942239 19658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37284a6220e4e3198e6f6e149a22233. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.932325 19511 tablet_service.cc:1467] Processing CreateTablet for tablet ffc97072515448ee8d301233977e6086 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250116 02:57:10.949484 19579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44eff29754e42feaafb330b0cfa2fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.950150 19508 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8cbd4766d94af88d1a276ec7f1ecc1 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250116 02:57:10.935503 19578 tablet_service.cc:1467] Processing CreateTablet for tablet 5604a103d76a4e8187cd388a9c94f99c (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250116 02:57:10.941608 19657 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8cbd4766d94af88d1a276ec7f1ecc1 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250116 02:57:10.963869 19507 tablet_service.cc:1467] Processing CreateTablet for tablet c722958228fb4aec8e0315ff6fc3595f (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250116 02:57:10.965939 19585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ba892d976584027b263e24ff2303c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.968587 19506 tablet_service.cc:1467] Processing CreateTablet for tablet 46904c6b96944fffa7561238601af1d9 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250116 02:57:10.967813 19656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c722958228fb4aec8e0315ff6fc3595f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.973430 19653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44eff29754e42feaafb330b0cfa2fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.968284 19509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37284a6220e4e3198e6f6e149a22233. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.974277 19657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cbd4766d94af88d1a276ec7f1ecc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.988107 19584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37284a6220e4e3198e6f6e149a22233. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.993117 19652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5604a103d76a4e8187cd388a9c94f99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.994796 19511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc97072515448ee8d301233977e6086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.994076 19504 tablet_service.cc:1467] Processing CreateTablet for tablet e44eff29754e42feaafb330b0cfa2fac (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250116 02:57:10.996342 19580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d32b1f88a2427d989bc38631930ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:10.987233 19505 tablet_service.cc:1467] Processing CreateTablet for tablet 10d32b1f88a2427d989bc38631930ad2 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
W20250116 02:57:10.997882 19466 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:57:11.008066 19578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5604a103d76a4e8187cd388a9c94f99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.011827 19503 tablet_service.cc:1467] Processing CreateTablet for tablet 5604a103d76a4e8187cd388a9c94f99c (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250116 02:57:10.984712 19654 tablet_service.cc:1467] Processing CreateTablet for tablet 10d32b1f88a2427d989bc38631930ad2 (DEFAULT_TABLE table=test-workload [id=0036284a217c40498bd4e863be65be27]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250116 02:57:11.015266 19581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46904c6b96944fffa7561238601af1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.020416 19654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d32b1f88a2427d989bc38631930ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.037525 19510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ba892d976584027b263e24ff2303c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.055814 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5604a103d76a4e8187cd388a9c94f99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.059751 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d32b1f88a2427d989bc38631930ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.075629 19504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44eff29754e42feaafb330b0cfa2fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.081897 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46904c6b96944fffa7561238601af1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.134912 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c722958228fb4aec8e0315ff6fc3595f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.157426 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cbd4766d94af88d1a276ec7f1ecc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:11.292933 19546 heartbeater.cc:501] Master 127.18.238.190:33651 was elected leader, sending a full tablet report...
I20250116 02:57:11.324120 19714 tablet_bootstrap.cc:492] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6: Bootstrap starting.
I20250116 02:57:11.336052 19713 tablet_bootstrap.cc:492] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7: Bootstrap starting.
I20250116 02:57:11.343461 19715 tablet_bootstrap.cc:492] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e: Bootstrap starting.
I20250116 02:57:11.351142 19715 tablet_bootstrap.cc:654] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.354835 19713 tablet_bootstrap.cc:654] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.363636 19715 tablet_bootstrap.cc:492] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e: No bootstrap required, opened a new log
I20250116 02:57:11.364324 19715 ts_tablet_manager.cc:1397] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.007s
I20250116 02:57:11.374934 19713 tablet_bootstrap.cc:492] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7: No bootstrap required, opened a new log
I20250116 02:57:11.377113 19713 ts_tablet_manager.cc:1397] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.011s
I20250116 02:57:11.389863 19713 raft_consensus.cc:357] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.391369 19713 raft_consensus.cc:383] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.391724 19713 raft_consensus.cc:738] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.392478 19713 consensus_queue.cc:260] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.406716 19714 tablet_bootstrap.cc:654] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.410988 19621 heartbeater.cc:501] Master 127.18.238.190:33651 was elected leader, sending a full tablet report...
I20250116 02:57:11.420153 19715 raft_consensus.cc:357] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.422163 19715 raft_consensus.cc:383] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.422461 19715 raft_consensus.cc:738] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.426395 19715 consensus_queue.cc:260] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.429248 19713 ts_tablet_manager.cc:1428] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7: Time spent starting tablet: real 0.050s user 0.008s sys 0.012s
I20250116 02:57:11.432658 19713 tablet_bootstrap.cc:492] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7: Bootstrap starting.
W20250116 02:57:11.440483 19622 tablet.cc:2367] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250116 02:57:11.443058 19714 tablet_bootstrap.cc:492] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6: No bootstrap required, opened a new log
I20250116 02:57:11.443652 19715 ts_tablet_manager.cc:1428] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent starting tablet: real 0.079s user 0.012s sys 0.004s
I20250116 02:57:11.443629 19714 ts_tablet_manager.cc:1397] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6: Time spent bootstrapping tablet: real 0.120s user 0.009s sys 0.009s
I20250116 02:57:11.445402 19715 tablet_bootstrap.cc:492] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e: Bootstrap starting.
I20250116 02:57:11.450886 19714 raft_consensus.cc:357] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.451790 19714 raft_consensus.cc:383] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.452128 19714 raft_consensus.cc:738] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.452311 19713 tablet_bootstrap.cc:654] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.453164 19714 consensus_queue.cc:260] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.456471 19715 tablet_bootstrap.cc:654] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.463436 19695 heartbeater.cc:501] Master 127.18.238.190:33651 was elected leader, sending a full tablet report...
I20250116 02:57:11.468665 19714 ts_tablet_manager.cc:1428] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6: Time spent starting tablet: real 0.022s user 0.013s sys 0.002s
I20250116 02:57:11.469844 19714 tablet_bootstrap.cc:492] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6: Bootstrap starting.
I20250116 02:57:11.470541 19715 tablet_bootstrap.cc:492] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e: No bootstrap required, opened a new log
I20250116 02:57:11.471093 19715 ts_tablet_manager.cc:1397] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.008s
I20250116 02:57:11.474529 19715 raft_consensus.cc:357] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.475745 19715 raft_consensus.cc:383] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.476200 19715 raft_consensus.cc:738] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.477017 19715 consensus_queue.cc:260] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.478292 19714 tablet_bootstrap.cc:654] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.480654 19715 ts_tablet_manager.cc:1428] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250116 02:57:11.481819 19715 tablet_bootstrap.cc:492] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e: Bootstrap starting.
I20250116 02:57:11.490119 19715 tablet_bootstrap.cc:654] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.496742 19714 tablet_bootstrap.cc:492] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6: No bootstrap required, opened a new log
I20250116 02:57:11.497447 19714 ts_tablet_manager.cc:1397] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.014s
I20250116 02:57:11.501425 19713 tablet_bootstrap.cc:492] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7: No bootstrap required, opened a new log
I20250116 02:57:11.502305 19713 ts_tablet_manager.cc:1397] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7: Time spent bootstrapping tablet: real 0.070s user 0.023s sys 0.023s
I20250116 02:57:11.503209 19714 raft_consensus.cc:357] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.505537 19714 raft_consensus.cc:383] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.505849 19714 raft_consensus.cc:738] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.505911 19713 raft_consensus.cc:357] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.506886 19713 raft_consensus.cc:383] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.506672 19714 consensus_queue.cc:260] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.507180 19713 raft_consensus.cc:738] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.509096 19713 consensus_queue.cc:260] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.511895 19713 ts_tablet_manager.cc:1428] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250116 02:57:11.509117 19714 ts_tablet_manager.cc:1428] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20250116 02:57:11.513929 19713 tablet_bootstrap.cc:492] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7: Bootstrap starting.
I20250116 02:57:11.522792 19715 tablet_bootstrap.cc:492] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e: No bootstrap required, opened a new log
I20250116 02:57:11.523391 19715 ts_tablet_manager.cc:1397] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.013s
I20250116 02:57:11.526288 19714 tablet_bootstrap.cc:492] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6: Bootstrap starting.
I20250116 02:57:11.527338 19715 raft_consensus.cc:357] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.528177 19715 raft_consensus.cc:383] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.528496 19715 raft_consensus.cc:738] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.529269 19715 consensus_queue.cc:260] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.534123 19715 ts_tablet_manager.cc:1428] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20250116 02:57:11.535427 19715 tablet_bootstrap.cc:492] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e: Bootstrap starting.
I20250116 02:57:11.537035 19714 tablet_bootstrap.cc:654] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.541428 19713 tablet_bootstrap.cc:654] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.543567 19715 tablet_bootstrap.cc:654] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.555608 19721 raft_consensus.cc:491] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.556260 19721 raft_consensus.cc:513] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.559374 19721 leader_election.cc:290] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:11.559824 19715 tablet_bootstrap.cc:492] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e: No bootstrap required, opened a new log
I20250116 02:57:11.560380 19715 ts_tablet_manager.cc:1397] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.012s
I20250116 02:57:11.561605 19713 tablet_bootstrap.cc:492] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7: No bootstrap required, opened a new log
I20250116 02:57:11.562165 19713 ts_tablet_manager.cc:1397] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.011s
I20250116 02:57:11.563431 19715 raft_consensus.cc:357] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.564281 19715 raft_consensus.cc:383] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.564599 19715 raft_consensus.cc:738] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.565312 19715 consensus_queue.cc:260] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.565450 19713 raft_consensus.cc:357] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.566334 19713 raft_consensus.cc:383] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.566656 19713 raft_consensus.cc:738] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.567947 19715 ts_tablet_manager.cc:1428] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250116 02:57:11.567695 19713 consensus_queue.cc:260] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.568986 19715 tablet_bootstrap.cc:492] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e: Bootstrap starting.
I20250116 02:57:11.570197 19713 ts_tablet_manager.cc:1428] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250116 02:57:11.571254 19713 tablet_bootstrap.cc:492] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7: Bootstrap starting.
I20250116 02:57:11.577165 19715 tablet_bootstrap.cc:654] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.578537 19713 tablet_bootstrap.cc:654] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.591708 19713 tablet_bootstrap.cc:492] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7: No bootstrap required, opened a new log
I20250116 02:57:11.592702 19713 ts_tablet_manager.cc:1397] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.005s
I20250116 02:57:11.594681 19715 tablet_bootstrap.cc:492] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e: No bootstrap required, opened a new log
I20250116 02:57:11.595252 19715 ts_tablet_manager.cc:1397] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.015s
I20250116 02:57:11.596058 19713 raft_consensus.cc:357] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.597051 19713 raft_consensus.cc:383] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.597342 19713 raft_consensus.cc:738] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.598054 19713 consensus_queue.cc:260] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.598433 19715 raft_consensus.cc:357] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.599331 19715 raft_consensus.cc:383] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.599648 19715 raft_consensus.cc:738] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.600709 19713 ts_tablet_manager.cc:1428] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250116 02:57:11.600327 19715 consensus_queue.cc:260] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.601969 19713 tablet_bootstrap.cc:492] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7: Bootstrap starting.
I20250116 02:57:11.603015 19715 ts_tablet_manager.cc:1428] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20250116 02:57:11.607817 19714 tablet_bootstrap.cc:492] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6: No bootstrap required, opened a new log
I20250116 02:57:11.608383 19714 ts_tablet_manager.cc:1397] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.003s
I20250116 02:57:11.609246 19713 tablet_bootstrap.cc:654] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.611534 19714 raft_consensus.cc:357] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:11.612445 19714 raft_consensus.cc:383] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.612751 19714 raft_consensus.cc:738] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.613428 19714 consensus_queue.cc:260] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:11.616272 19714 ts_tablet_manager.cc:1428] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20250116 02:57:11.617291 19714 tablet_bootstrap.cc:492] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6: Bootstrap starting.
I20250116 02:57:11.622426 19713 tablet_bootstrap.cc:492] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7: No bootstrap required, opened a new log
I20250116 02:57:11.623039 19713 ts_tablet_manager.cc:1397] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.003s
I20250116 02:57:11.624485 19714 tablet_bootstrap.cc:654] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.626128 19713 raft_consensus.cc:357] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:11.627125 19713 raft_consensus.cc:383] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.627526 19715 tablet_bootstrap.cc:492] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e: Bootstrap starting.
I20250116 02:57:11.627444 19713 raft_consensus.cc:738] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.628407 19713 consensus_queue.cc:260] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:11.631340 19713 ts_tablet_manager.cc:1428] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20250116 02:57:11.632429 19713 tablet_bootstrap.cc:492] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7: Bootstrap starting.
I20250116 02:57:11.635171 19715 tablet_bootstrap.cc:654] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.639636 19713 tablet_bootstrap.cc:654] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.652554 19713 tablet_bootstrap.cc:492] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7: No bootstrap required, opened a new log
I20250116 02:57:11.653136 19713 ts_tablet_manager.cc:1397] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20250116 02:57:11.653347 19715 tablet_bootstrap.cc:492] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e: No bootstrap required, opened a new log
I20250116 02:57:11.654429 19715 ts_tablet_manager.cc:1397] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.005s
I20250116 02:57:11.656394 19713 raft_consensus.cc:357] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.657287 19713 raft_consensus.cc:383] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.657611 19713 raft_consensus.cc:738] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.657660 19715 raft_consensus.cc:357] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.658560 19715 raft_consensus.cc:383] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.659066 19715 raft_consensus.cc:738] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.658578 19713 consensus_queue.cc:260] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.659864 19715 consensus_queue.cc:260] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.661911 19713 ts_tablet_manager.cc:1428] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7: Time spent starting tablet: real 0.008s user 0.003s sys 0.006s
I20250116 02:57:11.663092 19713 tablet_bootstrap.cc:492] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7: Bootstrap starting.
I20250116 02:57:11.667505 19715 ts_tablet_manager.cc:1428] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20250116 02:57:11.669972 19713 tablet_bootstrap.cc:654] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.672067 19715 tablet_bootstrap.cc:492] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e: Bootstrap starting.
I20250116 02:57:11.682228 19713 tablet_bootstrap.cc:492] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7: No bootstrap required, opened a new log
I20250116 02:57:11.682785 19713 ts_tablet_manager.cc:1397] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20250116 02:57:11.686089 19713 raft_consensus.cc:357] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.686878 19713 raft_consensus.cc:383] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.687165 19713 raft_consensus.cc:738] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.688257 19715 tablet_bootstrap.cc:654] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.687840 19713 consensus_queue.cc:260] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.691273 19713 ts_tablet_manager.cc:1428] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250116 02:57:11.692447 19713 tablet_bootstrap.cc:492] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7: Bootstrap starting.
I20250116 02:57:11.699684 19713 tablet_bootstrap.cc:654] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.707482 19715 tablet_bootstrap.cc:492] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e: No bootstrap required, opened a new log
I20250116 02:57:11.708056 19715 ts_tablet_manager.cc:1397] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.003s
I20250116 02:57:11.711174 19715 raft_consensus.cc:357] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.711922 19713 tablet_bootstrap.cc:492] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7: No bootstrap required, opened a new log
I20250116 02:57:11.711997 19715 raft_consensus.cc:383] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.712556 19713 ts_tablet_manager.cc:1397] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s
I20250116 02:57:11.712529 19715 raft_consensus.cc:738] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.713515 19715 consensus_queue.cc:260] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.715653 19713 raft_consensus.cc:357] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.716465 19713 raft_consensus.cc:383] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.716672 19715 ts_tablet_manager.cc:1428] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250116 02:57:11.716764 19713 raft_consensus.cc:738] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.717974 19715 tablet_bootstrap.cc:492] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e: Bootstrap starting.
I20250116 02:57:11.717634 19713 consensus_queue.cc:260] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.720919 19713 ts_tablet_manager.cc:1428] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20250116 02:57:11.721920 19713 tablet_bootstrap.cc:492] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7: Bootstrap starting.
I20250116 02:57:11.725589 19715 tablet_bootstrap.cc:654] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.728835 19713 tablet_bootstrap.cc:654] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.730793 19714 tablet_bootstrap.cc:492] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6: No bootstrap required, opened a new log
I20250116 02:57:11.731336 19714 ts_tablet_manager.cc:1397] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6: Time spent bootstrapping tablet: real 0.114s user 0.012s sys 0.029s
I20250116 02:57:11.734550 19714 raft_consensus.cc:357] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.735525 19714 raft_consensus.cc:383] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.735860 19714 raft_consensus.cc:738] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.736791 19714 consensus_queue.cc:260] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.739948 19714 ts_tablet_manager.cc:1428] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20250116 02:57:11.741443 19714 tablet_bootstrap.cc:492] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6: Bootstrap starting.
I20250116 02:57:11.744001 19713 tablet_bootstrap.cc:492] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7: No bootstrap required, opened a new log
I20250116 02:57:11.744542 19713 ts_tablet_manager.cc:1397] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.001s
I20250116 02:57:11.747370 19713 raft_consensus.cc:357] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.747977 19714 tablet_bootstrap.cc:654] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.748320 19713 raft_consensus.cc:383] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.748667 19713 raft_consensus.cc:738] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.749365 19713 consensus_queue.cc:260] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.751937 19713 ts_tablet_manager.cc:1428] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20250116 02:57:11.754209 19718 raft_consensus.cc:491] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.754956 19718 raft_consensus.cc:513] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.762754 19715 tablet_bootstrap.cc:492] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e: No bootstrap required, opened a new log
I20250116 02:57:11.762950 19718 leader_election.cc:290] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:11.763258 19715 ts_tablet_manager.cc:1397] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.008s
I20250116 02:57:11.767773 19715 raft_consensus.cc:357] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.768548 19715 raft_consensus.cc:383] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.768800 19715 raft_consensus.cc:738] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.769295 19715 consensus_queue.cc:260] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.786762 19721 raft_consensus.cc:491] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.787389 19721 raft_consensus.cc:513] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.790426 19720 raft_consensus.cc:491] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.792158 19715 ts_tablet_manager.cc:1428] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent starting tablet: real 0.029s user 0.005s sys 0.002s
I20250116 02:57:11.792709 19720 raft_consensus.cc:513] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.794256 19715 tablet_bootstrap.cc:492] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e: Bootstrap starting.
I20250116 02:57:11.816299 19718 raft_consensus.cc:491] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.820498 19718 raft_consensus.cc:513] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.822324 19714 tablet_bootstrap.cc:492] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6: No bootstrap required, opened a new log
I20250116 02:57:11.822906 19714 ts_tablet_manager.cc:1397] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6: Time spent bootstrapping tablet: real 0.082s user 0.014s sys 0.014s
I20250116 02:57:11.823053 19718 leader_election.cc:290] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:11.824368 19721 leader_election.cc:290] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:11.820518 19720 leader_election.cc:290] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:11.832974 19720 raft_consensus.cc:491] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.838501 19714 raft_consensus.cc:357] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.846279 19714 raft_consensus.cc:383] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.846673 19714 raft_consensus.cc:738] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.847470 19714 consensus_queue.cc:260] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.845225 19720 raft_consensus.cc:513] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.872515 19720 leader_election.cc:290] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:11.873148 19720 raft_consensus.cc:491] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.873607 19720 raft_consensus.cc:513] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.875943 19714 ts_tablet_manager.cc:1428] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6: Time spent starting tablet: real 0.053s user 0.007s sys 0.002s
I20250116 02:57:11.879959 19715 tablet_bootstrap.cc:654] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.880029 19720 leader_election.cc:290] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:11.880576 19720 raft_consensus.cc:491] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.885047 19714 tablet_bootstrap.cc:492] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6: Bootstrap starting.
I20250116 02:57:11.892818 19718 raft_consensus.cc:491] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.883138 19720 raft_consensus.cc:513] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.901172 19720 leader_election.cc:290] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:11.898701 19718 raft_consensus.cc:513] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.915409 19718 leader_election.cc:290] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:11.917591 19718 raft_consensus.cc:491] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.923800 19718 raft_consensus.cc:513] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.924484 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:11.925572 19596 raft_consensus.cc:2463] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 0.
I20250116 02:57:11.926764 19718 leader_election.cc:290] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:11.930863 19748 raft_consensus.cc:491] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.931562 19748 raft_consensus.cc:513] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.934690 19715 tablet_bootstrap.cc:492] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e: No bootstrap required, opened a new log
I20250116 02:57:11.935429 19715 ts_tablet_manager.cc:1397] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent bootstrapping tablet: real 0.141s user 0.012s sys 0.024s
I20250116 02:57:11.935238 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5604a103d76a4e8187cd388a9c94f99c" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:11.934258 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:11.937273 19595 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 0.
I20250116 02:57:11.945287 19487 leader_election.cc:304] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:11.938273 19596 raft_consensus.cc:2463] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 0.
I20250116 02:57:11.948060 19720 raft_consensus.cc:2798] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:11.952893 19720 raft_consensus.cc:491] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:11.952785 19521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:11.956108 19720 raft_consensus.cc:3054] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:11.949002 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:11.952349 19487 leader_election.cc:304] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:11.962319 19721 raft_consensus.cc:491] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:11.956007 19520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:11.965639 19721 raft_consensus.cc:513] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:11.968007 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:11.970811 19596 raft_consensus.cc:2463] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 0.
I20250116 02:57:11.971640 19721 leader_election.cc:290] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:11.967017 19519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:11.963761 19748 leader_election.cc:290] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:11.968815 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5604a103d76a4e8187cd388a9c94f99c" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:11.961422 19594 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 0.
I20250116 02:57:11.967294 19520 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 0.
I20250116 02:57:11.974359 19519 raft_consensus.cc:2463] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 0.
I20250116 02:57:11.967407 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:11.970180 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:11.974431 19595 raft_consensus.cc:2463] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 0.
I20250116 02:57:11.978768 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5604a103d76a4e8187cd388a9c94f99c" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:11.978956 19518 raft_consensus.cc:2463] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 0.
I20250116 02:57:11.978956 19517 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 0.
I20250116 02:57:11.983392 19562 leader_election.cc:304] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:11.984494 19718 raft_consensus.cc:2798] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:11.984936 19718 raft_consensus.cc:491] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:11.984604 19715 raft_consensus.cc:357] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:11.985592 19715 raft_consensus.cc:383] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:11.985972 19715 raft_consensus.cc:738] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Initialized, Role: FOLLOWER
I20250116 02:57:11.984982 19516 raft_consensus.cc:2463] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 0.
I20250116 02:57:11.986131 19718 raft_consensus.cc:3054] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:11.987927 19715 consensus_queue.cc:260] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:11.975898 19487 leader_election.cc:304] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:11.991438 19714 tablet_bootstrap.cc:654] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:11.994103 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:11.995554 19748 raft_consensus.cc:2798] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:11.995438 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:11.996027 19748 raft_consensus.cc:491] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:11.984584 19562 leader_election.cc:304] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:11.996466 19748 raft_consensus.cc:3054] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:11.995769 19516 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 0.
I20250116 02:57:11.996344 19595 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 0.
I20250116 02:57:11.992038 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:12.006590 19715 ts_tablet_manager.cc:1428] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e: Time spent starting tablet: real 0.071s user 0.005s sys 0.006s
W20250116 02:57:12.009797 19466 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:57:12.011544 19754 raft_consensus.cc:2798] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:11.998346 19756 raft_consensus.cc:491] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:12.008008 19487 leader_election.cc:304] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.007622 19596 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 0.
I20250116 02:57:12.007365 19636 leader_election.cc:304] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
I20250116 02:57:12.018896 19757 raft_consensus.cc:2798] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.020328 19721 raft_consensus.cc:2798] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.022212 19721 raft_consensus.cc:491] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.013255 19562 leader_election.cc:304] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.022557 19721 raft_consensus.cc:3054] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.019981 19757 raft_consensus.cc:491] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.025059 19757 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.017169 19756 raft_consensus.cc:513] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.028721 19756 leader_election.cc:290] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.029417 19756 raft_consensus.cc:2798] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.030113 19756 raft_consensus.cc:491] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.036955 19762 raft_consensus.cc:2798] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.037607 19762 raft_consensus.cc:491] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.038023 19762 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.022728 19636 leader_election.cc:304] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.030592 19756 raft_consensus.cc:3054] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.023223 19718 raft_consensus.cc:513] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.046955 19758 raft_consensus.cc:491] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:12.042797 19754 raft_consensus.cc:491] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.049258 19754 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.055133 19718 leader_election.cc:290] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.045279 19487 leader_election.cc:304] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.059084 19761 raft_consensus.cc:491] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:12.046943 19759 raft_consensus.cc:491] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:12.055647 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:12.047652 19758 raft_consensus.cc:513] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:12.055255 19720 raft_consensus.cc:513] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.072526 19714 tablet_bootstrap.cc:492] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6: No bootstrap required, opened a new log
I20250116 02:57:12.066893 19516 raft_consensus.cc:2463] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 0.
I20250116 02:57:12.077459 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.064535 19759 raft_consensus.cc:513] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:12.062165 19761 raft_consensus.cc:513] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.076161 19714 ts_tablet_manager.cc:1397] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6: Time spent bootstrapping tablet: real 0.191s user 0.009s sys 0.012s
I20250116 02:57:12.083287 19521 raft_consensus.cc:2388] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d5143dfed70340c3a0453c08ffd663c7 in current term 1: Already voted for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in this term.
I20250116 02:57:12.082358 19517 raft_consensus.cc:3054] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.085824 19720 leader_election.cc:290] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.103413 19562 leader_election.cc:304] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.104872 19718 raft_consensus.cc:2798] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.105252 19718 raft_consensus.cc:491] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.105872 19718 raft_consensus.cc:3054] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:57:12.355978 19700 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.18.238.190:33651, user_credentials={real_user=slave}} blocked reactor thread for 138522us
I20250116 02:57:12.356671 19721 raft_consensus.cc:513] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.151648 19754 raft_consensus.cc:513] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.358559 19721 leader_election.cc:290] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:12.358758 19754 leader_election.cc:290] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.359215 19721 raft_consensus.cc:2798] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.359364 19754 raft_consensus.cc:491] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:12.359985 19764 raft_consensus.cc:491] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:12.359840 19754 raft_consensus.cc:513] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.360373 19764 raft_consensus.cc:513] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.362123 19764 leader_election.cc:290] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.362234 19754 leader_election.cc:290] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.111449 19759 leader_election.cc:290] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:12.151255 19718 raft_consensus.cc:513] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.369796 19718 leader_election.cc:290] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.243687 19670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.370980 19517 raft_consensus.cc:2463] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 1.
W20250116 02:57:12.174007 19636 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:38935, user_credentials={real_user=slave}} blocked reactor thread for 151372us
I20250116 02:57:12.363345 19761 leader_election.cc:290] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.371912 19636 leader_election.cc:304] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.107496 19720 raft_consensus.cc:2798] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.376474 19720 raft_consensus.cc:491] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.376925 19720 raft_consensus.cc:3054] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.380985 19756 raft_consensus.cc:513] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
W20250116 02:57:12.138674 19562 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:38935, user_credentials={real_user=slave}} blocked reactor thread for 44571us
I20250116 02:57:12.105839 19748 raft_consensus.cc:513] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.362838 19721 raft_consensus.cc:491] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.387481 19763 raft_consensus.cc:2798] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.388206 19763 raft_consensus.cc:491] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.388659 19763 raft_consensus.cc:3054] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.297312 19669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5604a103d76a4e8187cd388a9c94f99c" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.374281 19668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.391196 19668 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 0.
I20250116 02:57:12.391535 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.392483 19596 raft_consensus.cc:2388] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c459e7a7a4648eca82c6d239d896df6 in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
I20250116 02:57:12.392867 19521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:12.393599 19521 raft_consensus.cc:2463] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 0.
I20250116 02:57:12.384961 19721 raft_consensus.cc:3054] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.397861 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.398831 19517 raft_consensus.cc:2388] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c459e7a7a4648eca82c6d239d896df6 in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
I20250116 02:57:12.400345 19714 raft_consensus.cc:357] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.401309 19714 raft_consensus.cc:383] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:12.400959 19762 raft_consensus.cc:513] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.393551 19757 raft_consensus.cc:513] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.401801 19714 raft_consensus.cc:738] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Initialized, Role: FOLLOWER
I20250116 02:57:12.401309 19748 leader_election.cc:290] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.408886 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.409742 19517 raft_consensus.cc:3054] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.410394 19562 leader_election.cc:304] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.411708 19761 raft_consensus.cc:2798] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:12.411872 19562 leader_election.cc:304] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.412269 19761 raft_consensus.cc:695] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:12.406589 19714 consensus_queue.cc:260] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.420825 19757 leader_election.cc:290] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.422389 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.407836 19670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.413224 19668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.420480 19718 raft_consensus.cc:2798] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.426852 19718 raft_consensus.cc:491] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.427314 19718 raft_consensus.cc:3054] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.421317 19756 leader_election.cc:290] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.413280 19761 consensus_queue.cc:237] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.428903 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.429692 19667 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 0.
I20250116 02:57:12.430296 19762 leader_election.cc:290] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.420554 19721 raft_consensus.cc:513] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.432228 19669 raft_consensus.cc:2388] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65a82ead3d34ffeb3ec16038d0a3e0e in current term 1: Already voted for candidate 0c459e7a7a4648eca82c6d239d896df6 in this term.
I20250116 02:57:12.435065 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.437388 19666 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.431274 19758 leader_election.cc:290] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839), f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935)
I20250116 02:57:12.440407 19670 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 0.
I20250116 02:57:12.443166 19720 raft_consensus.cc:513] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.443301 19763 raft_consensus.cc:513] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.447439 19521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.448311 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.449751 19668 raft_consensus.cc:2388] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65a82ead3d34ffeb3ec16038d0a3e0e in current term 1: Already voted for candidate 0c459e7a7a4648eca82c6d239d896df6 in this term.
I20250116 02:57:12.448333 19521 raft_consensus.cc:2388] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d5143dfed70340c3a0453c08ffd663c7 in current term 1: Already voted for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in this term.
I20250116 02:57:12.450634 19763 leader_election.cc:290] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:12.451539 19720 leader_election.cc:290] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.452066 19669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.453401 19669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.457890 19669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.459290 19714 ts_tablet_manager.cc:1428] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6: Time spent starting tablet: real 0.373s user 0.017s sys 0.001s
I20250116 02:57:12.459542 19669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.460438 19669 raft_consensus.cc:2388] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d5143dfed70340c3a0453c08ffd663c7 in current term 1: Already voted for candidate 0c459e7a7a4648eca82c6d239d896df6 in this term.
I20250116 02:57:12.460865 19714 tablet_bootstrap.cc:492] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6: Bootstrap starting.
I20250116 02:57:12.461158 19636 leader_election.cc:304] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6; no voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:12.461298 19669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.455392 19721 leader_election.cc:290] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:12.455037 19670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.465027 19662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
W20250116 02:57:12.469408 19559 leader_election.cc:343] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:12.469597 19769 raft_consensus.cc:491] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:12.469995 19559 leader_election.cc:304] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7; no voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:12.463438 19669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.462982 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.468931 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.473639 19521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.470431 19769 raft_consensus.cc:513] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:12.476279 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.482630 19520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:12.479045 19768 raft_consensus.cc:3054] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.479274 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.469309 19663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:12.472914 19665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5604a103d76a4e8187cd388a9c94f99c" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.487699 19665 raft_consensus.cc:2388] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f65a82ead3d34ffeb3ec16038d0a3e0e in current term 1: Already voted for candidate 0c459e7a7a4648eca82c6d239d896df6 in this term.
I20250116 02:57:12.486526 19521 raft_consensus.cc:2388] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d5143dfed70340c3a0453c08ffd663c7 in current term 1: Already voted for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in this term.
I20250116 02:57:12.485855 19517 raft_consensus.cc:2463] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 1.
I20250116 02:57:12.466890 19661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.490401 19662 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 0.
I20250116 02:57:12.491070 19771 raft_consensus.cc:2743] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.486342 19520 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 0.
W20250116 02:57:12.494403 19484 leader_election.cc:343] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:12.494350 19596 raft_consensus.cc:2388] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c459e7a7a4648eca82c6d239d896df6 in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
W20250116 02:57:12.496037 19484 leader_election.cc:343] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:12.496343 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5604a103d76a4e8187cd388a9c94f99c" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.496716 19484 leader_election.cc:304] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.497864 19720 raft_consensus.cc:2798] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.498741 19720 raft_consensus.cc:491] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.499097 19720 raft_consensus.cc:3054] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.492147 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5604a103d76a4e8187cd388a9c94f99c" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.500772 19518 raft_consensus.cc:2388] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c459e7a7a4648eca82c6d239d896df6 in current term 1: Already voted for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in this term.
W20250116 02:57:12.501611 19487 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:33723, user_credentials={real_user=slave}} blocked reactor thread for 456438us
I20250116 02:57:12.500773 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.502480 19664 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.503247 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.504832 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:12.483875 19718 raft_consensus.cc:513] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.488632 19669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.505407 19595 raft_consensus.cc:3054] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.507169 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.508752 19594 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 0.
I20250116 02:57:12.510214 19669 raft_consensus.cc:2388] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f65a82ead3d34ffeb3ec16038d0a3e0e in current term 1: Already voted for candidate 0c459e7a7a4648eca82c6d239d896df6 in this term.
I20250116 02:57:12.511904 19593 raft_consensus.cc:2388] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f65a82ead3d34ffeb3ec16038d0a3e0e in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
I20250116 02:57:12.513124 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:12.508293 19670 raft_consensus.cc:2388] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d5143dfed70340c3a0453c08ffd663c7 in current term 1: Already voted for candidate 0c459e7a7a4648eca82c6d239d896df6 in this term.
I20250116 02:57:12.515034 19636 leader_election.cc:304] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6; no voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e
W20250116 02:57:12.508029 19559 leader_election.cc:343] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:12.514773 19594 raft_consensus.cc:2388] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65a82ead3d34ffeb3ec16038d0a3e0e in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
I20250116 02:57:12.515456 19417 catalog_manager.cc:5526] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 1 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
W20250116 02:57:12.526000 19484 leader_election.cc:343] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:12.528314 19771 raft_consensus.cc:2743] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.526634 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:12.529407 19593 raft_consensus.cc:2388] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65a82ead3d34ffeb3ec16038d0a3e0e in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
I20250116 02:57:12.531265 19666 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 1.
I20250116 02:57:12.527451 19769 leader_election.cc:290] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:12.532466 19767 raft_consensus.cc:491] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:12.533085 19767 raft_consensus.cc:513] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.535775 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.536708 19594 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.533628 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5604a103d76a4e8187cd388a9c94f99c" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.547451 19718 leader_election.cc:290] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:12.547433 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.552029 19593 raft_consensus.cc:2388] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f65a82ead3d34ffeb3ec16038d0a3e0e in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
I20250116 02:57:12.549239 19487 leader_election.cc:304] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65a82ead3d34ffeb3ec16038d0a3e0e; no voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:12.549881 19767 leader_election.cc:290] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:12.553628 19762 raft_consensus.cc:2743] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.552254 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
W20250116 02:57:12.526755 19484 leader_election.cc:343] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:12.555930 19592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:12.557564 19592 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 0.
I20250116 02:57:12.548728 19720 raft_consensus.cc:513] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:12.549517 19559 leader_election.cc:304] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7; no voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:12.561040 19718 raft_consensus.cc:2743] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.562055 19559 leader_election.cc:304] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
I20250116 02:57:12.563051 19517 raft_consensus.cc:2388] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c459e7a7a4648eca82c6d239d896df6 in current term 1: Already voted for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in this term.
I20250116 02:57:12.566143 19768 raft_consensus.cc:2743] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.566700 19718 raft_consensus.cc:2798] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:12.567042 19718 raft_consensus.cc:491] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.567369 19718 raft_consensus.cc:3054] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.567584 19487 leader_election.cc:304] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65a82ead3d34ffeb3ec16038d0a3e0e; no voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:12.567646 19595 raft_consensus.cc:2463] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 1.
I20250116 02:57:12.570335 19664 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 1.
I20250116 02:57:12.574196 19762 raft_consensus.cc:3054] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.574962 19720 leader_election.cc:290] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Requested vote from peers 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:12.567967 19484 leader_election.cc:304] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters: d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:12.581532 19562 leader_election.cc:304] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7; no voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:12.583701 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
W20250116 02:57:12.580214 19559 leader_election.cc:343] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:12.586136 19768 raft_consensus.cc:2743] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.585901 19484 leader_election.cc:304] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.584298 19487 leader_election.cc:304] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65a82ead3d34ffeb3ec16038d0a3e0e; no voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:12.584627 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.588035 19748 raft_consensus.cc:2798] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:12.588797 19766 raft_consensus.cc:2743] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.588946 19748 raft_consensus.cc:695] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:12.581821 19762 raft_consensus.cc:2743] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.589939 19748 consensus_queue.cc:237] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.586182 19664 raft_consensus.cc:3054] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:12.581739 19720 raft_consensus.cc:2798] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:12.592957 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:12.594086 19714 tablet_bootstrap.cc:654] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:12.594368 19596 raft_consensus.cc:2463] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 1.
I20250116 02:57:12.593142 19720 raft_consensus.cc:695] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:12.601768 19636 leader_election.cc:304] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
W20250116 02:57:12.587077 19559 leader_election.cc:343] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:12.600203 19720 consensus_queue.cc:237] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.602674 19517 raft_consensus.cc:2463] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 1.
I20250116 02:57:12.615720 19487 leader_election.cc:304] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65a82ead3d34ffeb3ec16038d0a3e0e; no voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:12.616819 19759 raft_consensus.cc:2743] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.617271 19487 leader_election.cc:304] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters: 0c459e7a7a4648eca82c6d239d896df6
I20250116 02:57:12.616918 19664 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 1.
I20250116 02:57:12.618470 19759 raft_consensus.cc:2798] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:12.620898 19759 raft_consensus.cc:695] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:12.620213 19718 raft_consensus.cc:513] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:12.612232 19636 leader_election.cc:304] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters: f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:12.621853 19759 consensus_queue.cc:237] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.622915 19767 raft_consensus.cc:2758] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250116 02:57:12.622766 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:12.604880 19559 leader_election.cc:304] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters: 0c459e7a7a4648eca82c6d239d896df6
I20250116 02:57:12.624186 19484 leader_election.cc:304] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:12.628357 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.634997 19768 raft_consensus.cc:2798] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:12.635601 19768 raft_consensus.cc:695] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:12.636457 19768 consensus_queue.cc:237] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
W20250116 02:57:12.638921 19559 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:39839, user_credentials={real_user=slave}} blocked reactor thread for 51889.7us
I20250116 02:57:12.638970 19720 raft_consensus.cc:2798] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:12.632290 19769 raft_consensus.cc:2798] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:12.642426 19769 raft_consensus.cc:695] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 1 LEADER]: Becoming Leader. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Running, Role: LEADER
I20250116 02:57:12.642961 19595 raft_consensus.cc:2388] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f65a82ead3d34ffeb3ec16038d0a3e0e in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
I20250116 02:57:12.643496 19769 consensus_queue.cc:237] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.645205 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.641690 19720 raft_consensus.cc:695] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
W20250116 02:57:12.647022 19559 leader_election.cc:343] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:12.626562 19594 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 1.
W20250116 02:57:12.639475 19562 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:38935, user_credentials={real_user=slave}} blocked reactor thread for 52389.4us
I20250116 02:57:12.633334 19596 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 1.
I20250116 02:57:12.654893 19562 leader_election.cc:304] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters: 0c459e7a7a4648eca82c6d239d896df6
I20250116 02:57:12.656379 19761 raft_consensus.cc:2798] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:12.656975 19761 raft_consensus.cc:695] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:12.658819 19761 consensus_queue.cc:237] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.659684 19636 leader_election.cc:304] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters: f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:12.660614 19767 raft_consensus.cc:2798] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:57:12.660954 19767 raft_consensus.cc:491] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:12.661417 19767 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:12.648914 19720 consensus_queue.cc:237] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:12.662950 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:12.663776 19664 raft_consensus.cc:2388] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d5143dfed70340c3a0453c08ffd663c7 in current term 1: Already voted for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in this term.
I20250116 02:57:12.675035 19416 catalog_manager.cc:5526] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 0 to 1, leader changed from <none> to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:12.677803 19714 tablet_bootstrap.cc:492] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6: No bootstrap required, opened a new log
I20250116 02:57:12.678364 19714 ts_tablet_manager.cc:1397] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6: Time spent bootstrapping tablet: real 0.218s user 0.024s sys 0.024s
I20250116 02:57:12.680536 19767 raft_consensus.cc:513] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.681213 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.683012 19562 leader_election.cc:304] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7; no voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:12.687880 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:12.686398 19767 leader_election.cc:290] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 2 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:12.683135 19714 raft_consensus.cc:357] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.691470 19594 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:12.691975 19714 raft_consensus.cc:383] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:12.692332 19714 raft_consensus.cc:738] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Initialized, Role: FOLLOWER
I20250116 02:57:12.683694 19415 catalog_manager.cc:5526] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131). New cstate: current_term: 1 leader_uuid: "0c459e7a7a4648eca82c6d239d896df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:12.695539 19714 consensus_queue.cc:260] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.695416 19768 raft_consensus.cc:2743] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:57:12.673444 19718 leader_election.cc:290] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 1 election: Requested vote from peers 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839), f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935)
I20250116 02:57:12.687512 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:12.700309 19714 ts_tablet_manager.cc:1428] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20250116 02:57:12.714371 19594 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 2.
I20250116 02:57:12.714776 19714 tablet_bootstrap.cc:492] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6: Bootstrap starting.
I20250116 02:57:12.715847 19636 leader_election.cc:304] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters: f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:12.717222 19767 raft_consensus.cc:2798] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:12.718206 19767 raft_consensus.cc:695] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Becoming Leader. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Running, Role: LEADER
I20250116 02:57:12.719298 19767 consensus_queue.cc:237] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.741501 19714 tablet_bootstrap.cc:654] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:12.764303 19417 catalog_manager.cc:5526] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 0 to 1, leader changed from <none> to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:12.766428 19417 catalog_manager.cc:5526] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 0 to 1, leader changed from <none> to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:12.771234 19416 catalog_manager.cc:5526] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 1 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:12.773451 19416 catalog_manager.cc:5526] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 1 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:12.768385 19417 catalog_manager.cc:5526] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 0 to 1, leader changed from <none> to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:12.782691 19714 tablet_bootstrap.cc:492] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6: No bootstrap required, opened a new log
I20250116 02:57:12.783258 19714 ts_tablet_manager.cc:1397] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6: Time spent bootstrapping tablet: real 0.069s user 0.015s sys 0.009s
I20250116 02:57:12.785986 19714 raft_consensus.cc:357] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.786829 19714 raft_consensus.cc:383] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:12.787109 19714 raft_consensus.cc:738] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Initialized, Role: FOLLOWER
I20250116 02:57:12.793809 19714 consensus_queue.cc:260] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.796509 19714 ts_tablet_manager.cc:1428] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250116 02:57:12.797570 19714 tablet_bootstrap.cc:492] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6: Bootstrap starting.
I20250116 02:57:12.804631 19415 catalog_manager.cc:5526] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 reported cstate change: term changed from 1 to 2, leader changed from f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129) to 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131). New cstate: current_term: 2 leader_uuid: "0c459e7a7a4648eca82c6d239d896df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:12.812561 19714 tablet_bootstrap.cc:654] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:12.822186 19714 tablet_bootstrap.cc:492] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6: No bootstrap required, opened a new log
I20250116 02:57:12.822832 19714 ts_tablet_manager.cc:1397] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.002s
I20250116 02:57:12.826696 19714 raft_consensus.cc:357] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.827414 19714 raft_consensus.cc:383] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:12.827680 19714 raft_consensus.cc:738] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Initialized, Role: FOLLOWER
I20250116 02:57:12.828289 19714 consensus_queue.cc:260] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:12.828841 19718 consensus_queue.cc:1035] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:12.830518 19714 ts_tablet_manager.cc:1428] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250116 02:57:12.857185 19768 consensus_queue.cc:1035] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:13.012419 19767 raft_consensus.cc:491] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:13.013044 19767 raft_consensus.cc:513] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:13.015547 19767 leader_election.cc:290] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
W20250116 02:57:13.017416 19466 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:57:13.018137 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:13.018137 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:13.020175 19518 raft_consensus.cc:2388] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c459e7a7a4648eca82c6d239d896df6 in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
I20250116 02:57:13.021063 19636 leader_election.cc:304] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6; no voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:13.021745 19767 raft_consensus.cc:3054] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:13.026021 19767 raft_consensus.cc:2743] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:13.027220 19767 raft_consensus.cc:491] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:13.027626 19767 raft_consensus.cc:513] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:13.029100 19767 leader_election.cc:290] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:13.029940 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:13.030133 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:13.031041 19518 raft_consensus.cc:2388] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c459e7a7a4648eca82c6d239d896df6 in current term 1: Already voted for candidate d5143dfed70340c3a0453c08ffd663c7 in this term.
I20250116 02:57:13.032059 19636 leader_election.cc:304] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6; no voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:13.032899 19767 raft_consensus.cc:3054] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:13.038121 19767 raft_consensus.cc:2743] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:13.056044 19720 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:13.067332 19769 consensus_queue.cc:1035] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:13.090961 19766 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:13.092254 19769 consensus_queue.cc:1035] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:13.093251 19595 raft_consensus.cc:1235] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Rejecting Update request from peer f65a82ead3d34ffeb3ec16038d0a3e0e for earlier term 1. Current term is 2. Ops: []
I20250116 02:57:13.099663 19762 consensus_queue.cc:1046] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:13.100515 19762 raft_consensus.cc:3049] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:13.100817 19762 raft_consensus.cc:738] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:13.101886 19762 consensus_queue.cc:260] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:13.104174 19766 consensus_queue.cc:1035] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:13.105273 19762 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:13.105458 19766 consensus_queue.cc:1035] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:13.113703 19768 consensus_queue.cc:1035] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:13.117594 19718 consensus_queue.cc:1035] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:13.128150 19766 consensus_queue.cc:1035] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250116 02:57:13.138513 19718 consensus_queue.cc:1035] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:13.149834 19769 raft_consensus.cc:491] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:13.150599 19769 raft_consensus.cc:513] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:13.152824 19769 leader_election.cc:290] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:13.172187 19592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5143dfed70340c3a0453c08ffd663c7" is_pre_election: true
I20250116 02:57:13.168891 19520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:13.185971 19636 leader_election.cc:304] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6; no voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:13.187671 19769 raft_consensus.cc:2743] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:13.234110 19666 raft_consensus.cc:3054] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:13.240020 19768 consensus_queue.cc:1035] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:13.256457 19720 consensus_queue.cc:1035] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:13.260744 19517 raft_consensus.cc:1270] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250116 02:57:13.261273 19517 pending_rounds.cc:77] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e: Aborting all ops after (but not including) 0
I20250116 02:57:13.261629 19517 pending_rounds.cc:99] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250116 02:57:13.261940 19517 raft_consensus.cc:2883] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:57:13.263896 19769 consensus_queue.cc:1035] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:13.280783 19595 raft_consensus.cc:1270] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250116 02:57:13.282665 19771 consensus_queue.cc:1035] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250116 02:57:14.023900 19466 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:57:14.125487 19810 raft_consensus.cc:491] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:14.125941 19810 raft_consensus.cc:513] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.127525 19810 leader_election.cc:290] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:14.128482 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6" is_pre_election: true
I20250116 02:57:14.128430 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" is_pre_election: true
I20250116 02:57:14.129253 19518 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 1.
I20250116 02:57:14.129249 19666 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 1.
I20250116 02:57:14.130563 19562 leader_election.cc:304] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:14.131289 19810 raft_consensus.cc:2798] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:57:14.131705 19810 raft_consensus.cc:491] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:14.131995 19810 raft_consensus.cc:3054] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:14.139326 19810 raft_consensus.cc:513] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.141037 19810 leader_election.cc:290] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 2 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:14.142254 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:14.142269 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:14.143148 19518 raft_consensus.cc:3054] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:14.143328 19666 raft_consensus.cc:3054] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:14.148785 19518 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 2.
I20250116 02:57:14.148808 19666 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 2.
I20250116 02:57:14.150118 19559 leader_election.cc:304] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
I20250116 02:57:14.150887 19810 raft_consensus.cc:2798] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:14.151346 19810 raft_consensus.cc:695] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:14.152058 19810 consensus_queue.cc:237] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.161247 19415 catalog_manager.cc:5526] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 0 to 2, leader changed from <none> to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 2 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:14.589174 19595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ba892d976584027b263e24ff2303c74"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
mode: GRACEFUL
new_leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:35076
I20250116 02:57:14.589761 19595 raft_consensus.cc:604] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Received request to transfer leadership to TS f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:14.604772 19666 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233"
dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
mode: GRACEFUL
new_leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:46486
I20250116 02:57:14.605398 19666 raft_consensus.cc:604] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Received request to transfer leadership to TS f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:14.611080 19595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
mode: GRACEFUL
new_leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:35076
I20250116 02:57:14.611691 19595 raft_consensus.cc:604] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Received request to transfer leadership to TS f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:14.638528 19518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
mode: GRACEFUL
new_leader_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:56892
I20250116 02:57:14.639235 19518 raft_consensus.cc:604] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Received request to transfer leadership to TS d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:14.642946 19666 raft_consensus.cc:1270] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250116 02:57:14.644788 19810 consensus_queue.cc:1035] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:14.646325 19518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
mode: GRACEFUL
new_leader_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:56892
I20250116 02:57:14.649663 19518 raft_consensus.cc:604] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Received request to transfer leadership to TS d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:14.664171 19518 raft_consensus.cc:1270] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250116 02:57:14.667032 19824 consensus_queue.cc:1035] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:14.679466 19827 raft_consensus.cc:988] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6: : Instructing follower f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:14.679980 19808 raft_consensus.cc:1076] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Signalling peer f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:14.686899 19720 raft_consensus.cc:988] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e: : Instructing follower d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:14.687419 19720 raft_consensus.cc:1076] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Signalling peer d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:14.688015 19518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:54942
I20250116 02:57:14.688619 19518 raft_consensus.cc:491] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:14.688987 19518 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:14.690861 19595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:39354
I20250116 02:57:14.691426 19595 raft_consensus.cc:491] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:14.691996 19595 raft_consensus.cc:3054] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:14.695194 19518 raft_consensus.cc:513] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.697247 19518 leader_election.cc:290] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 3 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:14.698238 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:14.698484 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:14.698915 19596 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:14.699311 19666 raft_consensus.cc:3049] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:14.699105 19595 raft_consensus.cc:513] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.699611 19666 raft_consensus.cc:738] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Running, Role: LEADER
I20250116 02:57:14.700347 19666 consensus_queue.cc:260] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.701201 19595 leader_election.cc:290] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 2 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:14.701450 19666 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:14.702040 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:14.702242 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:14.702592 19518 raft_consensus.cc:3049] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:14.702759 19664 raft_consensus.cc:3054] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:14.702958 19518 raft_consensus.cc:738] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:14.703764 19518 consensus_queue.cc:260] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.704980 19518 raft_consensus.cc:3054] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:14.708196 19666 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 3.
I20250116 02:57:14.708806 19596 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 3.
I20250116 02:57:14.709517 19484 leader_election.cc:304] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:14.709800 19664 raft_consensus.cc:2463] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 2.
I20250116 02:57:14.710484 19828 raft_consensus.cc:2798] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:14.711043 19828 raft_consensus.cc:695] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:14.711181 19518 raft_consensus.cc:2463] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 2.
I20250116 02:57:14.711244 19559 leader_election.cc:304] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
I20250116 02:57:14.712110 19828 consensus_queue.cc:237] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.712755 19810 raft_consensus.cc:2798] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:14.713282 19810 raft_consensus.cc:695] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:14.714169 19810 consensus_queue.cc:237] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.721431 19415 catalog_manager.cc:5526] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 2 to 3, leader changed from 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131) to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 3 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:14.723701 19416 catalog_manager.cc:5526] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 1 to 2, leader changed from f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129) to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 2 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:14.786155 19824 raft_consensus.cc:988] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7: : Instructing follower f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:14.786723 19810 raft_consensus.cc:1076] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Signalling peer f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:14.788213 19518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ba892d976584027b263e24ff2303c74"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:54944
I20250116 02:57:14.788712 19518 raft_consensus.cc:491] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:14.788976 19518 raft_consensus.cc:3054] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:14.794914 19518 raft_consensus.cc:513] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.796720 19518 leader_election.cc:290] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 2 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:14.798002 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:14.798038 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:14.798583 19596 raft_consensus.cc:3049] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:14.798954 19666 raft_consensus.cc:3054] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:14.798954 19596 raft_consensus.cc:738] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:14.799840 19596 consensus_queue.cc:260] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.800894 19596 raft_consensus.cc:3054] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:14.804773 19666 raft_consensus.cc:2463] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 2.
I20250116 02:57:14.805969 19596 raft_consensus.cc:2463] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 2.
I20250116 02:57:14.806205 19484 leader_election.cc:304] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:14.807431 19828 raft_consensus.cc:2798] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:14.807906 19828 raft_consensus.cc:695] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:14.808714 19828 consensus_queue.cc:237] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:14.816730 19416 catalog_manager.cc:5526] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 1 to 2, leader changed from d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130) to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:15.015205 19720 raft_consensus.cc:988] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e: : Instructing follower d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:15.015688 19828 raft_consensus.cc:1076] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Signalling peer d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:15.017247 19596 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:39354
I20250116 02:57:15.017809 19596 raft_consensus.cc:491] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:15.018097 19596 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:15.023387 19596 raft_consensus.cc:513] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:15.025180 19596 leader_election.cc:290] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 2 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:15.026747 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:15.027223 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:15.027400 19518 raft_consensus.cc:3049] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:15.027956 19666 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:15.027894 19518 raft_consensus.cc:738] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:15.029136 19518 consensus_queue.cc:260] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:15.030511 19518 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:15.036756 19666 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 2.
I20250116 02:57:15.038261 19559 leader_election.cc:304] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
I20250116 02:57:15.038735 19518 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 2.
I20250116 02:57:15.039309 19810 raft_consensus.cc:2798] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:15.039916 19810 raft_consensus.cc:695] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:15.040761 19810 consensus_queue.cc:237] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:15.049010 19416 catalog_manager.cc:5526] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 1 to 2, leader changed from f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129) to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 2 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:15.070539 19824 raft_consensus.cc:988] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7: : Instructing follower f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:15.071141 19824 raft_consensus.cc:1076] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Signalling peer f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:15.072842 19518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:54944
I20250116 02:57:15.073388 19518 raft_consensus.cc:491] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:15.073684 19518 raft_consensus.cc:3054] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:15.079329 19518 raft_consensus.cc:513] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:15.081094 19518 leader_election.cc:290] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 2 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:15.082264 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:15.082266 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:15.083616 19596 raft_consensus.cc:3049] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:15.083686 19666 raft_consensus.cc:3054] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:15.084321 19596 raft_consensus.cc:738] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:15.085186 19596 consensus_queue.cc:260] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:15.086989 19596 raft_consensus.cc:3054] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:15.092905 19666 raft_consensus.cc:2463] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 2.
I20250116 02:57:15.094547 19484 leader_election.cc:304] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:15.094794 19596 raft_consensus.cc:2463] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 2.
I20250116 02:57:15.095497 19828 raft_consensus.cc:2798] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:15.095635 19666 raft_consensus.cc:1270] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:57:15.095973 19828 raft_consensus.cc:695] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:15.097183 19720 consensus_queue.cc:1035] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:15.097292 19828 consensus_queue.cc:237] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:15.111783 19596 raft_consensus.cc:1270] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:57:15.112702 19415 catalog_manager.cc:5526] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 1 to 2, leader changed from d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130) to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:15.113440 19839 consensus_queue.cc:1035] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:15.237823 19666 raft_consensus.cc:1270] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:15.239243 19821 consensus_queue.cc:1035] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:15.249246 19518 raft_consensus.cc:1270] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:15.250595 19821 consensus_queue.cc:1035] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:15.320163 19596 raft_consensus.cc:1270] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:15.321715 19839 consensus_queue.cc:1035] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:15.330438 19666 raft_consensus.cc:1270] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:15.331941 19720 consensus_queue.cc:1035] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:15.569746 19666 raft_consensus.cc:1270] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:15.571305 19720 consensus_queue.cc:1035] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:15.578346 19596 raft_consensus.cc:1270] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:15.580314 19720 consensus_queue.cc:1035] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:15.624516 19666 raft_consensus.cc:1270] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:15.625902 19821 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:15.628911 19517 raft_consensus.cc:1270] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:15.630481 19824 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:17.692433 19386 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20250116 02:57:17.693337 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:17.715860 19596 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
mode: GRACEFUL
new_leader_uuid: "0c459e7a7a4648eca82c6d239d896df6"
from {username='slave'} at 127.0.0.1:35082
I20250116 02:57:17.716403 19596 raft_consensus.cc:604] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Received request to transfer leadership to TS 0c459e7a7a4648eca82c6d239d896df6
I20250116 02:57:17.727615 19517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffc97072515448ee8d301233977e6086"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
mode: GRACEFUL
new_leader_uuid: "0c459e7a7a4648eca82c6d239d896df6"
from {username='slave'} at 127.0.0.1:56894
I20250116 02:57:17.728142 19517 raft_consensus.cc:604] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Received request to transfer leadership to TS 0c459e7a7a4648eca82c6d239d896df6
I20250116 02:57:17.729271 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250116 02:57:17.729748 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:18.094957 19849 raft_consensus.cc:988] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e: : Instructing follower 0c459e7a7a4648eca82c6d239d896df6 to start an election
I20250116 02:57:18.095468 19854 raft_consensus.cc:1076] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Signalling peer 0c459e7a7a4648eca82c6d239d896df6 to start an election
I20250116 02:57:18.097180 19666 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ffc97072515448ee8d301233977e6086"
dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
from {username='slave'} at 127.0.0.1:46478
I20250116 02:57:18.097736 19666 raft_consensus.cc:491] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:18.098042 19666 raft_consensus.cc:3054] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:18.103521 19666 raft_consensus.cc:513] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:18.105356 19666 leader_election.cc:290] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 2 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:18.106389 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:18.106564 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:18.107098 19517 raft_consensus.cc:3049] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:18.107252 19596 raft_consensus.cc:3054] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:18.107471 19517 raft_consensus.cc:738] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:18.108333 19517 consensus_queue.cc:260] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:18.109637 19517 raft_consensus.cc:3054] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:18.113437 19596 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 2.
I20250116 02:57:18.114794 19636 leader_election.cc:304] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
I20250116 02:57:18.115031 19517 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 2.
I20250116 02:57:18.115721 19808 raft_consensus.cc:2798] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:18.117002 19808 raft_consensus.cc:695] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Becoming Leader. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Running, Role: LEADER
I20250116 02:57:18.117895 19808 consensus_queue.cc:237] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:18.118988 19850 raft_consensus.cc:988] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7: : Instructing follower 0c459e7a7a4648eca82c6d239d896df6 to start an election
I20250116 02:57:18.119434 19821 raft_consensus.cc:1076] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Signalling peer 0c459e7a7a4648eca82c6d239d896df6 to start an election
I20250116 02:57:18.121176 19666 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2"
dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
from {username='slave'} at 127.0.0.1:41734
I20250116 02:57:18.122079 19666 raft_consensus.cc:491] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:18.122514 19666 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:18.126590 19417 catalog_manager.cc:5526] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 reported cstate change: term changed from 1 to 2, leader changed from f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129) to 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131). New cstate: current_term: 2 leader_uuid: "0c459e7a7a4648eca82c6d239d896df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:18.127806 19666 raft_consensus.cc:513] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:18.129503 19666 leader_election.cc:290] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 3 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:18.130923 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:18.131027 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:18.131784 19596 raft_consensus.cc:3049] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:18.131863 19517 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:18.132297 19596 raft_consensus.cc:738] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:18.133188 19596 consensus_queue.cc:260] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:18.134685 19596 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:18.137714 19517 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 3.
I20250116 02:57:18.138918 19636 leader_election.cc:304] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:18.139570 19808 raft_consensus.cc:2798] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:18.140004 19808 raft_consensus.cc:695] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 LEADER]: Becoming Leader. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Running, Role: LEADER
I20250116 02:57:18.140919 19808 consensus_queue.cc:237] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:18.141547 19596 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 3.
I20250116 02:57:18.149545 19417 catalog_manager.cc:5526] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 reported cstate change: term changed from 2 to 3, leader changed from d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130) to 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131). New cstate: current_term: 3 leader_uuid: "0c459e7a7a4648eca82c6d239d896df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:18.578056 19517 raft_consensus.cc:1270] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:18.579970 19808 consensus_queue.cc:1035] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:18.594667 19596 raft_consensus.cc:1270] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:18.597259 19808 consensus_queue.cc:1035] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:18.691848 19517 raft_consensus.cc:1270] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:18.693277 19808 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:57:18.704742 19596 raft_consensus.cc:1270] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:18.706470 19808 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:18.730970 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:18.736802 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:18.737375 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:19.738365 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:19.743383 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:19.743723 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:20.744663 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:20.749639 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:20.750083 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:21.751111 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:21.756585 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:21.757138 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:22.758054 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:22.763728 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:22.764114 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:23.764878 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:23.770684 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:23.771049 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:24.771946 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:24.778052 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:24.778528 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:25.779559 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:25.784844 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:25.785339 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:26.786154 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:26.791245 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:26.791630 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:27.828956 19386 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20250116 02:57:27.834596 19666 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffc97072515448ee8d301233977e6086"
dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
mode: GRACEFUL
new_leader_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:46486
I20250116 02:57:27.835384 19666 raft_consensus.cc:604] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Received request to transfer leadership to TS d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:27.841615 19517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ba892d976584027b263e24ff2303c74"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
mode: GRACEFUL
new_leader_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:56892
I20250116 02:57:27.842187 19517 raft_consensus.cc:604] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Received request to transfer leadership to TS d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:27.848268 19517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
mode: GRACEFUL
new_leader_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:56892
I20250116 02:57:27.848963 19517 raft_consensus.cc:604] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 LEADER]: Received request to transfer leadership to TS d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:27.854457 19517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
mode: GRACEFUL
new_leader_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:56892
I20250116 02:57:27.855048 19517 raft_consensus.cc:604] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Received request to transfer leadership to TS d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:27.855721 19883 raft_consensus.cc:988] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6: : Instructing follower d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:27.856191 19883 raft_consensus.cc:1076] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Signalling peer d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:27.858050 19595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ffc97072515448ee8d301233977e6086"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:39348
I20250116 02:57:27.858723 19595 raft_consensus.cc:491] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:27.859162 19595 raft_consensus.cc:3054] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:27.867197 19595 raft_consensus.cc:513] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:27.868592 19666 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2"
dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
mode: GRACEFUL
new_leader_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:46486
I20250116 02:57:27.869246 19666 raft_consensus.cc:604] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 LEADER]: Received request to transfer leadership to TS d5143dfed70340c3a0453c08ffd663c7
I20250116 02:57:27.870141 19595 leader_election.cc:290] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 3 election: Requested vote from peers 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839), f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935)
I20250116 02:57:27.871465 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:27.871806 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc97072515448ee8d301233977e6086" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:27.872243 19517 raft_consensus.cc:3054] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:27.872416 19666 raft_consensus.cc:3049] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:27.872855 19666 raft_consensus.cc:738] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Running, Role: LEADER
I20250116 02:57:27.873601 19666 consensus_queue.cc:260] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:27.874475 19666 raft_consensus.cc:3054] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:27.879556 19517 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 3.
I20250116 02:57:27.880842 19562 leader_election.cc:304] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:27.881821 19882 raft_consensus.cc:2798] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:27.881821 19666 raft_consensus.cc:2463] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 3.
I20250116 02:57:27.883718 19882 raft_consensus.cc:695] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 3 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:27.884819 19882 consensus_queue.cc:237] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } }
I20250116 02:57:27.887771 19854 raft_consensus.cc:988] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e: : Instructing follower d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:27.888286 19854 raft_consensus.cc:1076] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 LEADER]: Signalling peer d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:27.890216 19595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:39354
I20250116 02:57:27.890852 19595 raft_consensus.cc:491] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:27.891237 19595 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:27.898486 19595 raft_consensus.cc:513] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:27.900641 19595 leader_election.cc:290] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 4 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:27.900068 19417 catalog_manager.cc:5526] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 2 to 3, leader changed from 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131) to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 3 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:27.901638 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:27.901962 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:27.902305 19517 raft_consensus.cc:3049] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:57:27.902588 19666 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:27.902688 19517 raft_consensus.cc:738] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:27.903712 19517 consensus_queue.cc:260] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:27.904783 19517 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:27.911310 19666 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 4.
I20250116 02:57:27.911608 19517 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 4.
I20250116 02:57:27.912607 19559 leader_election.cc:304] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
I20250116 02:57:27.913549 19882 raft_consensus.cc:2798] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:57:27.914067 19882 raft_consensus.cc:695] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:27.915068 19882 consensus_queue.cc:237] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:27.927436 19417 catalog_manager.cc:5526] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 3 to 4, leader changed from f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129) to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 4 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:28.098311 19854 raft_consensus.cc:988] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e: : Instructing follower d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:28.098985 19885 raft_consensus.cc:1076] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Signalling peer d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:28.100661 19595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ba892d976584027b263e24ff2303c74"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:39354
I20250116 02:57:28.101188 19595 raft_consensus.cc:491] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:28.101469 19595 raft_consensus.cc:3054] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:28.107188 19595 raft_consensus.cc:513] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:28.108927 19595 leader_election.cc:290] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 3 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:28.109916 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:28.110061 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba892d976584027b263e24ff2303c74" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:28.110584 19517 raft_consensus.cc:3049] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:28.110641 19666 raft_consensus.cc:3054] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:28.111092 19517 raft_consensus.cc:738] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:28.111793 19517 consensus_queue.cc:260] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:28.112756 19517 raft_consensus.cc:3054] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:28.115762 19666 raft_consensus.cc:2463] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 3.
I20250116 02:57:28.116780 19559 leader_election.cc:304] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
I20250116 02:57:28.117574 19882 raft_consensus.cc:2798] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:28.118135 19882 raft_consensus.cc:695] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 3 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:28.119527 19517 raft_consensus.cc:2463] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 3.
I20250116 02:57:28.119062 19882 consensus_queue.cc:237] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:28.127130 19417 catalog_manager.cc:5526] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 2 to 3, leader changed from f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129) to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 3 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:28.134117 19854 raft_consensus.cc:988] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e: : Instructing follower d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:28.134786 19885 raft_consensus.cc:1076] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Signalling peer d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:28.136767 19595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:39354
I20250116 02:57:28.137424 19595 raft_consensus.cc:491] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:28.137706 19595 raft_consensus.cc:3054] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:28.143225 19595 raft_consensus.cc:513] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:28.144804 19595 leader_election.cc:290] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 3 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:28.145856 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:28.145998 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8cbd4766d94af88d1a276ec7f1ecc1" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:28.146667 19517 raft_consensus.cc:3049] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:28.146684 19664 raft_consensus.cc:3054] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:28.147142 19517 raft_consensus.cc:738] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:28.147913 19517 consensus_queue.cc:260] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:28.149255 19517 raft_consensus.cc:3054] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:28.155036 19664 raft_consensus.cc:2463] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 3.
I20250116 02:57:28.156441 19517 raft_consensus.cc:2463] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 3.
I20250116 02:57:28.156478 19559 leader_election.cc:304] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, d5143dfed70340c3a0453c08ffd663c7; no voters:
I20250116 02:57:28.157509 19882 raft_consensus.cc:2798] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:28.158056 19882 raft_consensus.cc:695] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 3 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:28.158950 19882 consensus_queue.cc:237] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:28.166419 19417 catalog_manager.cc:5526] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 2 to 3, leader changed from f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129) to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 3 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:28.240795 19874 raft_consensus.cc:988] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6: : Instructing follower d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:28.241266 19874 raft_consensus.cc:1076] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 LEADER]: Signalling peer d5143dfed70340c3a0453c08ffd663c7 to start an election
I20250116 02:57:28.242967 19595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
from {username='slave'} at 127.0.0.1:39348
I20250116 02:57:28.243471 19595 raft_consensus.cc:491] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:28.243743 19595 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:28.248699 19595 raft_consensus.cc:513] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:28.250279 19595 leader_election.cc:290] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 4 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:28.251446 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:28.251380 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "d5143dfed70340c3a0453c08ffd663c7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:28.252306 19517 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:28.252277 19664 raft_consensus.cc:3049] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:57:28.252795 19664 raft_consensus.cc:738] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Running, Role: LEADER
I20250116 02:57:28.253798 19664 consensus_queue.cc:260] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:28.255288 19664 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:28.260367 19517 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 4.
I20250116 02:57:28.261682 19562 leader_election.cc:304] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:28.262460 19882 raft_consensus.cc:2798] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:57:28.262387 19664 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5143dfed70340c3a0453c08ffd663c7 in term 4.
I20250116 02:57:28.263065 19882 raft_consensus.cc:695] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Becoming Leader. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:28.263830 19882 consensus_queue.cc:237] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:28.271418 19417 catalog_manager.cc:5526] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 reported cstate change: term changed from 3 to 4, leader changed from 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131) to d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130). New cstate: current_term: 4 leader_uuid: "d5143dfed70340c3a0453c08ffd663c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:28.293805 19664 raft_consensus.cc:1270] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:28.295145 19884 consensus_queue.cc:1035] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:28.307390 19517 raft_consensus.cc:1270] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:28.308727 19882 consensus_queue.cc:1035] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:57:28.311632 19518 raft_consensus.cc:1270] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 4 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250116 02:57:28.313063 19882 consensus_queue.cc:1035] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:28.326841 19664 raft_consensus.cc:1270] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 4 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250116 02:57:28.328806 19884 consensus_queue.cc:1035] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:57:28.540267 19664 raft_consensus.cc:1270] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:28.542325 19902 consensus_queue.cc:1035] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:28.563365 19518 raft_consensus.cc:1270] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:28.565335 19517 raft_consensus.cc:1270] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:28.566782 19884 consensus_queue.cc:1035] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:28.566918 19882 consensus_queue.cc:1035] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:28.580384 19664 raft_consensus.cc:1270] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:28.581990 19882 consensus_queue.cc:1035] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:28.656065 19664 raft_consensus.cc:1270] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 4 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:57:28.657461 19882 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:57:28.669368 19518 raft_consensus.cc:1270] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 4 FOLLOWER]: Refusing update from remote peer d5143dfed70340c3a0453c08ffd663c7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:57:28.670694 19882 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:57:30.912463 19386 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20250116 02:57:30.913426 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:30.920877 19595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
mode: GRACEFUL
new_leader_uuid: "0c459e7a7a4648eca82c6d239d896df6"
from {username='slave'} at 127.0.0.1:35082
I20250116 02:57:30.921592 19595 raft_consensus.cc:604] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Received request to transfer leadership to TS 0c459e7a7a4648eca82c6d239d896df6
I20250116 02:57:30.924609 19595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46904c6b96944fffa7561238601af1d9"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
mode: GRACEFUL
new_leader_uuid: "0c459e7a7a4648eca82c6d239d896df6"
from {username='slave'} at 127.0.0.1:35082
I20250116 02:57:30.925258 19595 raft_consensus.cc:604] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Received request to transfer leadership to TS 0c459e7a7a4648eca82c6d239d896df6
I20250116 02:57:30.927804 19595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
mode: GRACEFUL
new_leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:35082
I20250116 02:57:30.928321 19595 raft_consensus.cc:604] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Received request to transfer leadership to TS f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:30.931030 19595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
mode: GRACEFUL
new_leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:35082
I20250116 02:57:30.931524 19595 raft_consensus.cc:604] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Received request to transfer leadership to TS f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:30.933655 19595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac"
dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
mode: GRACEFUL
new_leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:35082
I20250116 02:57:30.934128 19595 raft_consensus.cc:604] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Received request to transfer leadership to TS f65a82ead3d34ffeb3ec16038d0a3e0e
I20250116 02:57:30.935098 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250116 02:57:30.935568 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:30.983512 19902 raft_consensus.cc:988] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7: : Instructing follower f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:30.983947 19902 raft_consensus.cc:1076] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Signalling peer f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:30.985488 19517 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:54944
I20250116 02:57:30.985989 19517 raft_consensus.cc:491] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:30.986272 19517 raft_consensus.cc:3054] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:30.991246 19517 raft_consensus.cc:513] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:30.993659 19517 leader_election.cc:290] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 3 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:30.994809 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:30.995121 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44eff29754e42feaafb330b0cfa2fac" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:30.995441 19595 raft_consensus.cc:3049] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:30.995769 19595 raft_consensus.cc:738] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:30.995786 19664 raft_consensus.cc:3054] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:30.996589 19595 consensus_queue.cc:260] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:30.997697 19595 raft_consensus.cc:3054] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:31.001829 19664 raft_consensus.cc:2463] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 3.
I20250116 02:57:31.003014 19595 raft_consensus.cc:2463] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 3.
I20250116 02:57:31.003348 19484 leader_election.cc:304] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:31.005152 19913 raft_consensus.cc:2798] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:31.006012 19913 raft_consensus.cc:695] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:31.006994 19913 consensus_queue.cc:237] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.015388 19417 catalog_manager.cc:5526] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 2 to 3, leader changed from d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130) to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 3 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:31.284984 19912 raft_consensus.cc:988] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7: : Instructing follower 0c459e7a7a4648eca82c6d239d896df6 to start an election
I20250116 02:57:31.285424 19902 raft_consensus.cc:1076] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Signalling peer 0c459e7a7a4648eca82c6d239d896df6 to start an election
I20250116 02:57:31.286849 19664 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46904c6b96944fffa7561238601af1d9"
dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
from {username='slave'} at 127.0.0.1:41734
I20250116 02:57:31.287297 19664 raft_consensus.cc:491] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:31.287544 19664 raft_consensus.cc:3054] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:31.291792 19664 raft_consensus.cc:513] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.293891 19664 leader_election.cc:290] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 3 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:31.294979 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:31.295151 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46904c6b96944fffa7561238601af1d9" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:31.295516 19517 raft_consensus.cc:3054] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:31.295794 19595 raft_consensus.cc:3049] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:31.296146 19595 raft_consensus.cc:738] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:31.296974 19595 consensus_queue.cc:260] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.298018 19595 raft_consensus.cc:3054] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:31.300694 19517 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 3.
I20250116 02:57:31.301942 19636 leader_election.cc:304] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:31.302839 19874 raft_consensus.cc:2798] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:31.302862 19595 raft_consensus.cc:2463] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 3.
I20250116 02:57:31.304456 19874 raft_consensus.cc:695] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 LEADER]: Becoming Leader. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Running, Role: LEADER
I20250116 02:57:31.305428 19874 consensus_queue.cc:237] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.313974 19417 catalog_manager.cc:5526] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 reported cstate change: term changed from 2 to 3, leader changed from d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130) to 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131). New cstate: current_term: 3 leader_uuid: "0c459e7a7a4648eca82c6d239d896df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:31.388070 19912 raft_consensus.cc:988] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7: : Instructing follower f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:31.389461 19912 raft_consensus.cc:1076] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Signalling peer f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:31.390292 19922 raft_consensus.cc:988] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7: : Instructing follower f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:31.390825 19912 raft_consensus.cc:1076] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Signalling peer f65a82ead3d34ffeb3ec16038d0a3e0e to start an election
I20250116 02:57:31.391273 19518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:54944
I20250116 02:57:31.391876 19518 raft_consensus.cc:491] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:31.392365 19518 raft_consensus.cc:3054] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:31.392585 19517 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233"
dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
from {username='slave'} at 127.0.0.1:54944
I20250116 02:57:31.393275 19517 raft_consensus.cc:491] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:31.393637 19517 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:57:31.400759 19517 raft_consensus.cc:513] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.400759 19518 raft_consensus.cc:513] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.403045 19518 leader_election.cc:290] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 2 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:31.403049 19517 leader_election.cc:290] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 5 election: Requested vote from peers d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723), 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131:39839)
I20250116 02:57:31.404318 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:31.404917 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37284a6220e4e3198e6f6e149a22233" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:31.405277 19595 raft_consensus.cc:3049] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:57:31.405582 19595 raft_consensus.cc:738] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:31.405623 19666 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:57:31.404917 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:31.406666 19595 consensus_queue.cc:260] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.408070 19595 raft_consensus.cc:3054] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:57:31.408906 19670 raft_consensus.cc:1270] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:31.407246 19596 raft_consensus.cc:3049] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:31.410976 19596 raft_consensus.cc:738] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:31.412062 19922 raft_consensus.cc:988] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7: : Instructing follower 0c459e7a7a4648eca82c6d239d896df6 to start an election
I20250116 02:57:31.411689 19596 consensus_queue.cc:260] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.412607 19912 raft_consensus.cc:1076] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Signalling peer 0c459e7a7a4648eca82c6d239d896df6 to start an election
I20250116 02:57:31.413015 19666 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 5.
I20250116 02:57:31.413285 19596 raft_consensus.cc:3054] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:31.415055 19484 leader_election.cc:304] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:31.414960 19913 consensus_queue.cc:1035] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:31.415762 19595 raft_consensus.cc:2463] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 5.
I20250116 02:57:31.416253 19914 raft_consensus.cc:2798] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:57:31.404317 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722958228fb4aec8e0315ff6fc3595f" candidate_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
I20250116 02:57:31.416816 19914 raft_consensus.cc:695] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 5 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:31.416839 19664 raft_consensus.cc:3054] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:31.417894 19914 consensus_queue.cc:237] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.423362 19664 raft_consensus.cc:2463] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 2.
I20250116 02:57:31.423362 19596 raft_consensus.cc:2463] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65a82ead3d34ffeb3ec16038d0a3e0e in term 2.
I20250116 02:57:31.425755 19487 leader_election.cc:304] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5143dfed70340c3a0453c08ffd663c7, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:31.426690 19914 raft_consensus.cc:2798] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:31.427412 19914 raft_consensus.cc:695] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Becoming Leader. State: Replica: f65a82ead3d34ffeb3ec16038d0a3e0e, State: Running, Role: LEADER
I20250116 02:57:31.426255 19666 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2"
dest_uuid: "0c459e7a7a4648eca82c6d239d896df6"
from {username='slave'} at 127.0.0.1:41734
I20250116 02:57:31.428095 19666 raft_consensus.cc:491] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:31.428490 19666 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:57:31.428316 19914 consensus_queue.cc:237] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.432283 19417 catalog_manager.cc:5526] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 4 to 5, leader changed from d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130) to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 5 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:31.435904 19666 raft_consensus.cc:513] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.438091 19666 leader_election.cc:290] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 5 election: Requested vote from peers f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935), d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130:33723)
I20250116 02:57:31.443286 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e"
I20250116 02:57:31.443836 19518 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:57:31.447402 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d32b1f88a2427d989bc38631930ad2" candidate_uuid: "0c459e7a7a4648eca82c6d239d896df6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d5143dfed70340c3a0453c08ffd663c7"
I20250116 02:57:31.448084 19596 raft_consensus.cc:3049] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:57:31.448387 19596 raft_consensus.cc:738] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d5143dfed70340c3a0453c08ffd663c7, State: Running, Role: LEADER
I20250116 02:57:31.449143 19596 consensus_queue.cc:260] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.450392 19596 raft_consensus.cc:3054] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:57:31.451581 19518 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 5.
I20250116 02:57:31.451081 19417 catalog_manager.cc:5526] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e reported cstate change: term changed from 1 to 2, leader changed from d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130) to f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:31.452745 19636 leader_election.cc:304] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c459e7a7a4648eca82c6d239d896df6, f65a82ead3d34ffeb3ec16038d0a3e0e; no voters:
I20250116 02:57:31.453563 19874 raft_consensus.cc:2798] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:57:31.454102 19874 raft_consensus.cc:695] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 5 LEADER]: Becoming Leader. State: Replica: 0c459e7a7a4648eca82c6d239d896df6, State: Running, Role: LEADER
I20250116 02:57:31.454972 19874 consensus_queue.cc:237] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } }
I20250116 02:57:31.456429 19595 raft_consensus.cc:1270] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:31.458377 19929 consensus_queue.cc:1035] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:31.462237 19596 raft_consensus.cc:2463] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c459e7a7a4648eca82c6d239d896df6 in term 5.
I20250116 02:57:31.468626 19415 catalog_manager.cc:5526] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 reported cstate change: term changed from 4 to 5, leader changed from d5143dfed70340c3a0453c08ffd663c7 (127.18.238.130) to 0c459e7a7a4648eca82c6d239d896df6 (127.18.238.131). New cstate: current_term: 5 leader_uuid: "0c459e7a7a4648eca82c6d239d896df6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:31.765661 19595 raft_consensus.cc:1270] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:57:31.766878 19874 consensus_queue.cc:1035] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:31.780421 19518 raft_consensus.cc:1270] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:57:31.781870 19874 consensus_queue.cc:1035] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:31.839897 19666 raft_consensus.cc:1270] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 5 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250116 02:57:31.841310 19929 consensus_queue.cc:1035] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:57:31.850788 19595 raft_consensus.cc:1270] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 5 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250116 02:57:31.852694 19913 consensus_queue.cc:1035] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:57:31.932955 19666 raft_consensus.cc:1270] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:31.934273 19913 consensus_queue.cc:1035] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c459e7a7a4648eca82c6d239d896df6" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:31.936537 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:31.945132 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:31.945768 19595 raft_consensus.cc:1270] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Refusing update from remote peer f65a82ead3d34ffeb3ec16038d0a3e0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:31.945941 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:31.947291 19913 consensus_queue.cc:1035] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:32.047094 19595 raft_consensus.cc:1270] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 5 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:57:32.048520 19874 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5143dfed70340c3a0453c08ffd663c7" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:57:32.057016 19518 raft_consensus.cc:1270] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 5 FOLLOWER]: Refusing update from remote peer 0c459e7a7a4648eca82c6d239d896df6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:57:32.059170 19874 consensus_queue.cc:1035] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65a82ead3d34ffeb3ec16038d0a3e0e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:57:32.947161 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:32.951949 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:32.952436 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:33.953584 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:33.959944 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:33.960422 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:34.961539 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:34.966792 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:34.967315 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:35.968251 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:35.973320 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:35.973879 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:36.974803 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:36.980290 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:36.980744 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:37.981710 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:37.986982 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:37.987425 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:38.988435 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:38.995024 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:38.995548 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:39.996688 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:40.002040 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:40.002506 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:41.032665 19386 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20250116 02:57:41.033334 19386 tablet_server.cc:178] TabletServer@127.18.238.129:0 shutting down...
I20250116 02:57:41.067958 19386 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250116 02:57:41.068827 19386 tablet_replica.cc:331] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e: stopping tablet replica
I20250116 02:57:41.069605 19386 raft_consensus.cc:2238] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.070580 19386 raft_consensus.cc:2267] T e44eff29754e42feaafb330b0cfa2fac P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.073452 19386 tablet_replica.cc:331] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e: stopping tablet replica
I20250116 02:57:41.074259 19386 raft_consensus.cc:2238] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.074836 19386 raft_consensus.cc:2267] T 2ba892d976584027b263e24ff2303c74 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.077046 19386 tablet_replica.cc:331] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e: stopping tablet replica
I20250116 02:57:41.077759 19386 raft_consensus.cc:2238] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.078261 19386 raft_consensus.cc:2267] T ffc97072515448ee8d301233977e6086 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.080847 19386 tablet_replica.cc:331] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e: stopping tablet replica
I20250116 02:57:41.081619 19386 raft_consensus.cc:2238] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.082443 19386 raft_consensus.cc:2267] T c722958228fb4aec8e0315ff6fc3595f P f65a82ead3d34ffeb3ec16038d0a3e0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.084336 19386 tablet_replica.cc:331] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e: stopping tablet replica
I20250116 02:57:41.084877 19386 raft_consensus.cc:2238] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.085276 19386 raft_consensus.cc:2267] T 5604a103d76a4e8187cd388a9c94f99c P f65a82ead3d34ffeb3ec16038d0a3e0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.087126 19386 tablet_replica.cc:331] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e: stopping tablet replica
I20250116 02:57:41.087682 19386 raft_consensus.cc:2238] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.088070 19386 raft_consensus.cc:2267] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.089771 19386 tablet_replica.cc:331] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e: stopping tablet replica
I20250116 02:57:41.090225 19386 raft_consensus.cc:2238] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 5 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.090664 19386 raft_consensus.cc:2267] T 10d32b1f88a2427d989bc38631930ad2 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 5 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.092417 19386 tablet_replica.cc:331] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e: stopping tablet replica
I20250116 02:57:41.092878 19386 raft_consensus.cc:2238] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 5 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.093636 19386 raft_consensus.cc:2267] T d37284a6220e4e3198e6f6e149a22233 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 5 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.095252 19386 tablet_replica.cc:331] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e: stopping tablet replica
I20250116 02:57:41.095887 19386 raft_consensus.cc:2238] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.096552 19386 raft_consensus.cc:2267] T 46904c6b96944fffa7561238601af1d9 P f65a82ead3d34ffeb3ec16038d0a3e0e [term 3 FOLLOWER]: Raft consensus is shut down!
W20250116 02:57:41.195715 19636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.129:38935: connect: Connection refused (error 111)
I20250116 02:57:41.199000 19386 tablet_server.cc:195] TabletServer@127.18.238.129:0 shutdown complete.
W20250116 02:57:41.201457 19636 consensus_peers.cc:487] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 -> Peer f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935): Couldn't send request to peer f65a82ead3d34ffeb3ec16038d0a3e0e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:38935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250116 02:57:41.202461 19562 consensus_peers.cc:487] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 -> Peer f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935): Couldn't send request to peer f65a82ead3d34ffeb3ec16038d0a3e0e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:38935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250116 02:57:41.222142 19386 tablet_server.cc:178] TabletServer@127.18.238.130:0 shutting down...
W20250116 02:57:41.246575 19636 consensus_peers.cc:487] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 -> Peer f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935): Couldn't send request to peer f65a82ead3d34ffeb3ec16038d0a3e0e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:38935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250116 02:57:41.254637 19386 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250116 02:57:41.255401 19386 tablet_replica.cc:331] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7: stopping tablet replica
I20250116 02:57:41.256055 19386 raft_consensus.cc:2238] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 3 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.257128 19386 raft_consensus.cc:2267] T 2ba892d976584027b263e24ff2303c74 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.259199 19386 tablet_replica.cc:331] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7: stopping tablet replica
I20250116 02:57:41.260016 19386 raft_consensus.cc:2238] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.260558 19386 raft_consensus.cc:2267] T c722958228fb4aec8e0315ff6fc3595f P d5143dfed70340c3a0453c08ffd663c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.262493 19386 tablet_replica.cc:331] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7: stopping tablet replica
I20250116 02:57:41.263134 19386 raft_consensus.cc:2238] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 3 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.264065 19386 raft_consensus.cc:2267] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.265921 19386 tablet_replica.cc:331] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7: stopping tablet replica
I20250116 02:57:41.266459 19386 raft_consensus.cc:2238] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.266940 19386 raft_consensus.cc:2267] T 5604a103d76a4e8187cd388a9c94f99c P d5143dfed70340c3a0453c08ffd663c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.268771 19386 tablet_replica.cc:331] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7: stopping tablet replica
I20250116 02:57:41.269290 19386 raft_consensus.cc:2238] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 3 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.270145 19386 raft_consensus.cc:2267] T ffc97072515448ee8d301233977e6086 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.272150 19386 tablet_replica.cc:331] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7: stopping tablet replica
I20250116 02:57:41.272712 19386 raft_consensus.cc:2238] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.273206 19386 raft_consensus.cc:2267] T 46904c6b96944fffa7561238601af1d9 P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.275298 19386 tablet_replica.cc:331] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7: stopping tablet replica
I20250116 02:57:41.275852 19386 raft_consensus.cc:2238] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.276283 19386 raft_consensus.cc:2267] T d37284a6220e4e3198e6f6e149a22233 P d5143dfed70340c3a0453c08ffd663c7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.278786 19386 tablet_replica.cc:331] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7: stopping tablet replica
I20250116 02:57:41.279374 19386 raft_consensus.cc:2238] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.279963 19386 raft_consensus.cc:2267] T 10d32b1f88a2427d989bc38631930ad2 P d5143dfed70340c3a0453c08ffd663c7 [term 5 FOLLOWER]: Raft consensus is shut down!
W20250116 02:57:41.281769 19636 consensus_peers.cc:487] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 -> Peer f65a82ead3d34ffeb3ec16038d0a3e0e (127.18.238.129:38935): Couldn't send request to peer f65a82ead3d34ffeb3ec16038d0a3e0e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:38935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250116 02:57:41.282029 19386 tablet_replica.cc:331] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7: stopping tablet replica
I20250116 02:57:41.282660 19386 raft_consensus.cc:2238] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.283115 19386 raft_consensus.cc:2267] T e44eff29754e42feaafb330b0cfa2fac P d5143dfed70340c3a0453c08ffd663c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.315354 19386 tablet_server.cc:195] TabletServer@127.18.238.130:0 shutdown complete.
I20250116 02:57:41.335575 19386 tablet_server.cc:178] TabletServer@127.18.238.131:0 shutting down...
I20250116 02:57:41.373620 19386 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250116 02:57:41.374456 19386 tablet_replica.cc:331] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6: stopping tablet replica
I20250116 02:57:41.375315 19386 raft_consensus.cc:2238] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.375855 19386 raft_consensus.cc:2267] T e44eff29754e42feaafb330b0cfa2fac P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.378022 19386 tablet_replica.cc:331] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6: stopping tablet replica
I20250116 02:57:41.378590 19386 raft_consensus.cc:2238] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 5 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.379590 19386 raft_consensus.cc:2267] T 10d32b1f88a2427d989bc38631930ad2 P 0c459e7a7a4648eca82c6d239d896df6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.381429 19386 tablet_replica.cc:331] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6: stopping tablet replica
I20250116 02:57:41.381948 19386 raft_consensus.cc:2238] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.382421 19386 raft_consensus.cc:2267] T d37284a6220e4e3198e6f6e149a22233 P 0c459e7a7a4648eca82c6d239d896df6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.384516 19386 tablet_replica.cc:331] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6: stopping tablet replica
I20250116 02:57:41.385126 19386 raft_consensus.cc:2238] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.386183 19386 raft_consensus.cc:2267] T 46904c6b96944fffa7561238601af1d9 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.388113 19386 tablet_replica.cc:331] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6: stopping tablet replica
I20250116 02:57:41.388700 19386 raft_consensus.cc:2238] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.389117 19386 raft_consensus.cc:2267] T 6f8cbd4766d94af88d1a276ec7f1ecc1 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.390863 19386 tablet_replica.cc:331] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6: stopping tablet replica
I20250116 02:57:41.391309 19386 raft_consensus.cc:2238] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.391664 19386 raft_consensus.cc:2267] T c722958228fb4aec8e0315ff6fc3595f P 0c459e7a7a4648eca82c6d239d896df6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.393365 19386 tablet_replica.cc:331] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6: stopping tablet replica
I20250116 02:57:41.393810 19386 raft_consensus.cc:2238] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 1 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.394529 19386 raft_consensus.cc:2267] T 5604a103d76a4e8187cd388a9c94f99c P 0c459e7a7a4648eca82c6d239d896df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.396238 19386 tablet_replica.cc:331] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6: stopping tablet replica
I20250116 02:57:41.396703 19386 raft_consensus.cc:2238] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.397053 19386 raft_consensus.cc:2267] T ffc97072515448ee8d301233977e6086 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.399132 19386 tablet_replica.cc:331] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6: stopping tablet replica
I20250116 02:57:41.399632 19386 raft_consensus.cc:2238] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:57:41.400116 19386 raft_consensus.cc:2267] T 2ba892d976584027b263e24ff2303c74 P 0c459e7a7a4648eca82c6d239d896df6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.429678 19386 tablet_server.cc:195] TabletServer@127.18.238.131:0 shutdown complete.
I20250116 02:57:41.450302 19386 master.cc:537] Master@127.18.238.190:33651 shutting down...
I20250116 02:57:41.672264 19386 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 1 LEADER]: Raft consensus shutting down.
I20250116 02:57:41.672914 19386 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:57:41.673203 19386 tablet_replica.cc:331] T 00000000000000000000000000000000 P 534a60b857234a638acfeeff7be83bda: stopping tablet replica
I20250116 02:57:41.694993 19386 master.cc:559] Master@127.18.238.190:33651 shutdown complete.
0c459e7a7a4648eca82c6d239d896df6 1
d5143dfed70340c3a0453c08ffd663c7 4
f65a82ead3d34ffeb3ec16038d0a3e0e 4
0c459e7a7a4648eca82c6d239d896df6 3
d5143dfed70340c3a0453c08ffd663c7 3
f65a82ead3d34ffeb3ec16038d0a3e0e 3
0c459e7a7a4648eca82c6d239d896df6 1
d5143dfed70340c3a0453c08ffd663c7 8
f65a82ead3d34ffeb3ec16038d0a3e0e 0
0c459e7a7a4648eca82c6d239d896df6 3
d5143dfed70340c3a0453c08ffd663c7 3
f65a82ead3d34ffeb3ec16038d0a3e0e 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (33298 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250116 02:57:41.737629 19386 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.238.190:42057
I20250116 02:57:41.739063 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:57:41.745716 19954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:41.746021 19955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:41.747051 19957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:41.748521 19386 server_base.cc:1034] running on GCE node
I20250116 02:57:41.749465 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:57:41.749663 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:57:41.749794 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996261749782 us; error 0 us; skew 500 ppm
I20250116 02:57:41.750300 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:57:41.759189 19386 webserver.cc:458] Webserver started at http://127.18.238.190:38159/ using document root <none> and password file <none>
I20250116 02:57:41.759764 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:57:41.759953 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:57:41.760212 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:57:41.761574 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/master-0-root/instance:
uuid: "7f03230ee9634460b5cd609ad1391208"
format_stamp: "Formatted at 2025-01-16 02:57:41 on dist-test-slave-qn1x"
I20250116 02:57:41.766958 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20250116 02:57:41.770912 19962 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:41.771819 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250116 02:57:41.772141 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/master-0-root
uuid: "7f03230ee9634460b5cd609ad1391208"
format_stamp: "Formatted at 2025-01-16 02:57:41 on dist-test-slave-qn1x"
I20250116 02:57:41.772431 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:57:41.791393 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:57:41.792663 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:57:41.840128 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.190:42057
I20250116 02:57:41.840237 20013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.190:42057 every 8 connection(s)
I20250116 02:57:41.844527 20014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:41.856500 20014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208: Bootstrap starting.
I20250116 02:57:41.861099 20014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:41.866322 20014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208: No bootstrap required, opened a new log
I20250116 02:57:41.868990 20014 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f03230ee9634460b5cd609ad1391208" member_type: VOTER }
I20250116 02:57:41.869480 20014 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:41.869711 20014 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f03230ee9634460b5cd609ad1391208, State: Initialized, Role: FOLLOWER
I20250116 02:57:41.870514 20014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f03230ee9634460b5cd609ad1391208" member_type: VOTER }
I20250116 02:57:41.871035 20014 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250116 02:57:41.871266 20014 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250116 02:57:41.871531 20014 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:41.876894 20014 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f03230ee9634460b5cd609ad1391208" member_type: VOTER }
I20250116 02:57:41.877635 20014 leader_election.cc:304] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f03230ee9634460b5cd609ad1391208; no voters:
I20250116 02:57:41.878989 20014 leader_election.cc:290] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250116 02:57:41.879359 20017 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:41.880798 20017 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 1 LEADER]: Becoming Leader. State: Replica: 7f03230ee9634460b5cd609ad1391208, State: Running, Role: LEADER
I20250116 02:57:41.881613 20017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f03230ee9634460b5cd609ad1391208" member_type: VOTER }
I20250116 02:57:41.882207 20014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [sys.catalog]: configured and running, proceeding with master startup.
I20250116 02:57:41.886931 20019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f03230ee9634460b5cd609ad1391208. Latest consensus state: current_term: 1 leader_uuid: "7f03230ee9634460b5cd609ad1391208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f03230ee9634460b5cd609ad1391208" member_type: VOTER } }
I20250116 02:57:41.887704 20019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [sys.catalog]: This master's current role is: LEADER
I20250116 02:57:41.889451 20018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f03230ee9634460b5cd609ad1391208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f03230ee9634460b5cd609ad1391208" member_type: VOTER } }
I20250116 02:57:41.890012 20018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [sys.catalog]: This master's current role is: LEADER
I20250116 02:57:41.891950 20024 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250116 02:57:41.900249 20024 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250116 02:57:41.902927 19386 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250116 02:57:41.910697 20024 catalog_manager.cc:1348] Generated new cluster ID: da52c8cf82be405a91f16aa6eef46d31
I20250116 02:57:41.911098 20024 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250116 02:57:41.928838 20024 catalog_manager.cc:1371] Generated new certificate authority record
I20250116 02:57:41.930135 20024 catalog_manager.cc:1505] Loading token signing keys...
I20250116 02:57:41.950042 20024 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208: Generated new TSK 0
I20250116 02:57:41.950824 20024 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250116 02:57:41.970093 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:57:41.977998 20035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:41.978713 20036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:41.982743 19386 server_base.cc:1034] running on GCE node
W20250116 02:57:41.983114 20038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:41.984512 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:57:41.984726 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:57:41.984858 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996261984847 us; error 0 us; skew 500 ppm
I20250116 02:57:41.985370 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:57:41.988117 19386 webserver.cc:458] Webserver started at http://127.18.238.129:34263/ using document root <none> and password file <none>
I20250116 02:57:41.988663 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:57:41.988847 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:57:41.989079 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:57:41.990386 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-0-root/instance:
uuid: "ec07b35db1e64604a8285de3d895b36e"
format_stamp: "Formatted at 2025-01-16 02:57:41 on dist-test-slave-qn1x"
I20250116 02:57:41.995566 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20250116 02:57:41.999764 20043 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:42.000697 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250116 02:57:42.001031 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-0-root
uuid: "ec07b35db1e64604a8285de3d895b36e"
format_stamp: "Formatted at 2025-01-16 02:57:41 on dist-test-slave-qn1x"
I20250116 02:57:42.001363 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:57:42.018908 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:57:42.020859 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:57:42.023296 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:57:42.026057 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:57:42.026293 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:42.026549 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:57:42.026763 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250116 02:57:42.078532 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.129:35441
I20250116 02:57:42.078696 20105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.129:35441 every 8 connection(s)
I20250116 02:57:42.084643 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:57:42.095114 20110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:42.097504 20113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:42.097961 20111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:42.101224 19386 server_base.cc:1034] running on GCE node
I20250116 02:57:42.101337 20106 heartbeater.cc:346] Connected to a master server at 127.18.238.190:42057
I20250116 02:57:42.101801 20106 heartbeater.cc:463] Registering TS with master...
I20250116 02:57:42.102384 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:57:42.102653 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:57:42.102639 20106 heartbeater.cc:509] Master 127.18.238.190:42057 requested a full tablet report, sending...
I20250116 02:57:42.102990 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996262102972 us; error 0 us; skew 500 ppm
I20250116 02:57:42.104512 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:57:42.106456 19979 ts_manager.cc:194] Registered new tserver with Master: ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441)
I20250116 02:57:42.108100 19386 webserver.cc:458] Webserver started at http://127.18.238.130:40109/ using document root <none> and password file <none>
I20250116 02:57:42.108573 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:57:42.108497 19979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45710
I20250116 02:57:42.108777 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:57:42.109153 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:57:42.110448 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-1-root/instance:
uuid: "28eb102bb0fe457fbe61295f0e9e967a"
format_stamp: "Formatted at 2025-01-16 02:57:42 on dist-test-slave-qn1x"
I20250116 02:57:42.115296 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250116 02:57:42.119621 20118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:42.120539 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250116 02:57:42.120884 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-1-root
uuid: "28eb102bb0fe457fbe61295f0e9e967a"
format_stamp: "Formatted at 2025-01-16 02:57:42 on dist-test-slave-qn1x"
I20250116 02:57:42.121209 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:57:42.136373 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:57:42.137902 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:57:42.139626 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:57:42.142213 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:57:42.142488 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:42.142771 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:57:42.142928 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:42.190021 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.130:46575
I20250116 02:57:42.190136 20180 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.130:46575 every 8 connection(s)
I20250116 02:57:42.196039 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:57:42.204782 20184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:42.205312 20185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:57:42.207423 20187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:57:42.208734 19386 server_base.cc:1034] running on GCE node
I20250116 02:57:42.209766 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:57:42.209985 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:57:42.210155 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996262210136 us; error 0 us; skew 500 ppm
I20250116 02:57:42.210538 20181 heartbeater.cc:346] Connected to a master server at 127.18.238.190:42057
I20250116 02:57:42.210870 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:57:42.211038 20181 heartbeater.cc:463] Registering TS with master...
I20250116 02:57:42.211999 20181 heartbeater.cc:509] Master 127.18.238.190:42057 requested a full tablet report, sending...
I20250116 02:57:42.214349 19386 webserver.cc:458] Webserver started at http://127.18.238.131:38295/ using document root <none> and password file <none>
I20250116 02:57:42.215009 19979 ts_manager.cc:194] Registered new tserver with Master: 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:42.215286 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:57:42.215571 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:57:42.215924 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:57:42.217077 19979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45726
I20250116 02:57:42.217727 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-2-root/instance:
uuid: "d4721f3cbce642dc82cf320fe38a5c24"
format_stamp: "Formatted at 2025-01-16 02:57:42 on dist-test-slave-qn1x"
I20250116 02:57:42.223822 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250116 02:57:42.227861 20192 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:42.228740 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250116 02:57:42.229084 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-2-root
uuid: "d4721f3cbce642dc82cf320fe38a5c24"
format_stamp: "Formatted at 2025-01-16 02:57:42 on dist-test-slave-qn1x"
I20250116 02:57:42.229378 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736996228253955-19386-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:57:42.240545 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:57:42.241981 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:57:42.243494 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:57:42.245816 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:57:42.246019 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:42.246248 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:57:42.246407 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:57:42.293946 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.131:33605
I20250116 02:57:42.294081 20254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.131:33605 every 8 connection(s)
I20250116 02:57:42.310963 20255 heartbeater.cc:346] Connected to a master server at 127.18.238.190:42057
I20250116 02:57:42.311337 20255 heartbeater.cc:463] Registering TS with master...
I20250116 02:57:42.312096 20255 heartbeater.cc:509] Master 127.18.238.190:42057 requested a full tablet report, sending...
I20250116 02:57:42.314000 19979 ts_manager.cc:194] Registered new tserver with Master: d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:42.314352 19386 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014412351s
I20250116 02:57:42.315368 19979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45742
I20250116 02:57:42.317395 19386 test_util.cc:274] Using random seed: -181334732
I20250116 02:57:42.343657 19979 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45750:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250116 02:57:42.347054 19979 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250116 02:57:42.443553 20220 tablet_service.cc:1467] Processing CreateTablet for tablet 7c54e2afff19422a81f41473cb9ae916 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250116 02:57:42.445253 20220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c54e2afff19422a81f41473cb9ae916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.444871 20070 tablet_service.cc:1467] Processing CreateTablet for tablet 8de1db99051645bd97c9e3fb8c65232c (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250116 02:57:42.446406 20070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de1db99051645bd97c9e3fb8c65232c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.443419 20071 tablet_service.cc:1467] Processing CreateTablet for tablet 7c54e2afff19422a81f41473cb9ae916 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250116 02:57:42.452539 20219 tablet_service.cc:1467] Processing CreateTablet for tablet 8de1db99051645bd97c9e3fb8c65232c (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250116 02:57:42.447491 20064 tablet_service.cc:1467] Processing CreateTablet for tablet 8b150dc2209849c38456b6fb80e4ff53 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250116 02:57:42.456449 20219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de1db99051645bd97c9e3fb8c65232c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.456866 20062 tablet_service.cc:1467] Processing CreateTablet for tablet db9804b2c38548ba84e86f7a058a1829 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250116 02:57:42.457747 20071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c54e2afff19422a81f41473cb9ae916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.457423 20216 tablet_service.cc:1467] Processing CreateTablet for tablet acc97c03c06d4af2b969daa6c856f153 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250116 02:57:42.458357 20062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9804b2c38548ba84e86f7a058a1829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.459085 20216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc97c03c06d4af2b969daa6c856f153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.454054 20065 tablet_service.cc:1467] Processing CreateTablet for tablet c5119b4436ca4ae7b847cb5a33148178 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250116 02:57:42.452749 20063 tablet_service.cc:1467] Processing CreateTablet for tablet 744b14b25ab74319abb56a19e21a2d2c (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250116 02:57:42.462353 20146 tablet_service.cc:1467] Processing CreateTablet for tablet 7c54e2afff19422a81f41473cb9ae916 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250116 02:57:42.453593 20218 tablet_service.cc:1467] Processing CreateTablet for tablet 388b9f9b579d4f0ab9758daf60f3a680 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250116 02:57:42.463984 20146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c54e2afff19422a81f41473cb9ae916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.464359 20218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388b9f9b579d4f0ab9758daf60f3a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.464187 20143 tablet_service.cc:1467] Processing CreateTablet for tablet 827733d443b142479360c6e9dbf5483b (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250116 02:57:42.452957 20066 tablet_service.cc:1467] Processing CreateTablet for tablet 9a755ec1e74245599302dfc6394e26e4 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250116 02:57:42.455016 20217 tablet_service.cc:1467] Processing CreateTablet for tablet 827733d443b142479360c6e9dbf5483b (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250116 02:57:42.467175 20142 tablet_service.cc:1467] Processing CreateTablet for tablet acc97c03c06d4af2b969daa6c856f153 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250116 02:57:42.467458 20217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827733d443b142479360c6e9dbf5483b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.468509 20144 tablet_service.cc:1467] Processing CreateTablet for tablet 388b9f9b579d4f0ab9758daf60f3a680 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250116 02:57:42.470064 20141 tablet_service.cc:1467] Processing CreateTablet for tablet 9a755ec1e74245599302dfc6394e26e4 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250116 02:57:42.450316 20068 tablet_service.cc:1467] Processing CreateTablet for tablet 827733d443b142479360c6e9dbf5483b (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250116 02:57:42.471073 20145 tablet_service.cc:1467] Processing CreateTablet for tablet 8de1db99051645bd97c9e3fb8c65232c (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250116 02:57:42.451584 20067 tablet_service.cc:1467] Processing CreateTablet for tablet acc97c03c06d4af2b969daa6c856f153 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250116 02:57:42.466167 20137 tablet_service.cc:1467] Processing CreateTablet for tablet db9804b2c38548ba84e86f7a058a1829 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250116 02:57:42.472316 20140 tablet_service.cc:1467] Processing CreateTablet for tablet c5119b4436ca4ae7b847cb5a33148178 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250116 02:57:42.459123 20064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b150dc2209849c38456b6fb80e4ff53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.473738 20066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a755ec1e74245599302dfc6394e26e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.449008 20069 tablet_service.cc:1467] Processing CreateTablet for tablet 388b9f9b579d4f0ab9758daf60f3a680 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250116 02:57:42.475059 20065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5119b4436ca4ae7b847cb5a33148178. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.474665 20139 tablet_service.cc:1467] Processing CreateTablet for tablet 8b150dc2209849c38456b6fb80e4ff53 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250116 02:57:42.475808 20068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827733d443b142479360c6e9dbf5483b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.465605 20143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827733d443b142479360c6e9dbf5483b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.477280 20138 tablet_service.cc:1467] Processing CreateTablet for tablet 744b14b25ab74319abb56a19e21a2d2c (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250116 02:57:42.479851 20215 tablet_service.cc:1467] Processing CreateTablet for tablet 9a755ec1e74245599302dfc6394e26e4 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250116 02:57:42.481354 20215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a755ec1e74245599302dfc6394e26e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.482421 20063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744b14b25ab74319abb56a19e21a2d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.483523 20142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc97c03c06d4af2b969daa6c856f153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.483956 20067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc97c03c06d4af2b969daa6c856f153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.485527 20145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de1db99051645bd97c9e3fb8c65232c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.486975 20141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a755ec1e74245599302dfc6394e26e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.487603 20140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5119b4436ca4ae7b847cb5a33148178. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.488349 20144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388b9f9b579d4f0ab9758daf60f3a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.489055 20139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b150dc2209849c38456b6fb80e4ff53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.491060 20137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9804b2c38548ba84e86f7a058a1829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.495010 20138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744b14b25ab74319abb56a19e21a2d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.501237 20069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388b9f9b579d4f0ab9758daf60f3a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.521449 20214 tablet_service.cc:1467] Processing CreateTablet for tablet c5119b4436ca4ae7b847cb5a33148178 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250116 02:57:42.523857 20214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5119b4436ca4ae7b847cb5a33148178. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.528414 20213 tablet_service.cc:1467] Processing CreateTablet for tablet 8b150dc2209849c38456b6fb80e4ff53 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250116 02:57:42.530045 20213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b150dc2209849c38456b6fb80e4ff53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.561522 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 744b14b25ab74319abb56a19e21a2d2c (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250116 02:57:42.573035 20211 tablet_service.cc:1467] Processing CreateTablet for tablet db9804b2c38548ba84e86f7a058a1829 (DEFAULT_TABLE table=test-workload [id=8a75e985e2fb4ad39162c7b33a4f328c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250116 02:57:42.563704 20212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744b14b25ab74319abb56a19e21a2d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.601125 20211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9804b2c38548ba84e86f7a058a1829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:57:42.653821 20270 tablet_bootstrap.cc:492] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:42.659570 20271 tablet_bootstrap.cc:492] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:42.666311 20272 tablet_bootstrap.cc:492] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:42.676728 20270 tablet_bootstrap.cc:654] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.682963 20271 tablet_bootstrap.cc:654] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.692833 20272 tablet_bootstrap.cc:654] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.700920 20271 tablet_bootstrap.cc:492] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:42.701596 20271 ts_tablet_manager.cc:1397] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.011s
I20250116 02:57:42.705622 20271 raft_consensus.cc:357] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.706538 20271 raft_consensus.cc:383] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.706904 20271 raft_consensus.cc:738] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.707676 20271 consensus_queue.cc:260] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.714514 20255 heartbeater.cc:501] Master 127.18.238.190:42057 was elected leader, sending a full tablet report...
I20250116 02:57:42.714838 20272 tablet_bootstrap.cc:492] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:42.715106 20271 ts_tablet_manager.cc:1428] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20250116 02:57:42.715251 20270 tablet_bootstrap.cc:492] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:42.715605 20272 ts_tablet_manager.cc:1397] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.007s
I20250116 02:57:42.715812 20270 ts_tablet_manager.cc:1397] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.063s user 0.016s sys 0.011s
I20250116 02:57:42.716363 20271 tablet_bootstrap.cc:492] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:42.718477 20272 raft_consensus.cc:357] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.719276 20272 raft_consensus.cc:383] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.719553 20272 raft_consensus.cc:738] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.719972 20270 raft_consensus.cc:357] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.720719 20270 raft_consensus.cc:383] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.720310 20272 consensus_queue.cc:260] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.721009 20270 raft_consensus.cc:738] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.721714 20270 consensus_queue.cc:260] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.723613 20271 tablet_bootstrap.cc:654] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.740701 20272 ts_tablet_manager.cc:1428] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.025s user 0.012s sys 0.011s
I20250116 02:57:42.741575 20270 ts_tablet_manager.cc:1428] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.025s user 0.019s sys 0.004s
I20250116 02:57:42.742480 20181 heartbeater.cc:501] Master 127.18.238.190:42057 was elected leader, sending a full tablet report...
I20250116 02:57:42.742579 20106 heartbeater.cc:501] Master 127.18.238.190:42057 was elected leader, sending a full tablet report...
I20250116 02:57:42.743216 20270 tablet_bootstrap.cc:492] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:42.743829 20271 tablet_bootstrap.cc:492] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:42.744342 20271 ts_tablet_manager.cc:1397] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20250116 02:57:42.747645 20271 raft_consensus.cc:357] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.752177 20271 raft_consensus.cc:383] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.752617 20271 raft_consensus.cc:738] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.753547 20271 consensus_queue.cc:260] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.756173 20271 ts_tablet_manager.cc:1428] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20250116 02:57:42.757103 20271 tablet_bootstrap.cc:492] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:42.750378 20270 tablet_bootstrap.cc:654] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.751624 20272 tablet_bootstrap.cc:492] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:42.767146 20270 tablet_bootstrap.cc:492] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:42.767796 20270 ts_tablet_manager.cc:1397] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.001s
I20250116 02:57:42.768113 20276 raft_consensus.cc:491] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:42.768657 20276 raft_consensus.cc:513] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.769987 20272 tablet_bootstrap.cc:654] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.768682 20271 tablet_bootstrap.cc:654] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.771116 20270 raft_consensus.cc:357] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.773844 20270 raft_consensus.cc:383] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.774362 20270 raft_consensus.cc:738] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.775151 20270 consensus_queue.cc:260] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.778165 20270 ts_tablet_manager.cc:1428] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20250116 02:57:42.779415 20270 tablet_bootstrap.cc:492] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:42.785858 20278 raft_consensus.cc:491] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:42.786320 20270 tablet_bootstrap.cc:654] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.787600 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:42.786520 20278 raft_consensus.cc:513] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.790545 20278 leader_election.cc:290] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
W20250116 02:57:42.795542 20196 leader_election.cc:343] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:42.801167 20276 leader_election.cc:290] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:42.801533 20270 tablet_bootstrap.cc:492] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:42.802345 20270 ts_tablet_manager.cc:1397] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s
W20250116 02:57:42.803860 20256 tablet.cc:2367] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250116 02:57:42.806017 20270 raft_consensus.cc:357] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.806916 20270 raft_consensus.cc:383] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.807241 20271 tablet_bootstrap.cc:492] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:42.807432 20270 raft_consensus.cc:738] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.807799 20271 ts_tablet_manager.cc:1397] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.051s user 0.023s sys 0.006s
I20250116 02:57:42.808351 20270 consensus_queue.cc:260] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.811034 20270 ts_tablet_manager.cc:1428] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250116 02:57:42.810654 20271 raft_consensus.cc:357] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.811403 20271 raft_consensus.cc:383] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.811713 20271 raft_consensus.cc:738] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.812031 20270 tablet_bootstrap.cc:492] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:42.812383 20271 consensus_queue.cc:260] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.818797 20270 tablet_bootstrap.cc:654] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.828771 20271 ts_tablet_manager.cc:1428] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20250116 02:57:42.833241 20271 tablet_bootstrap.cc:492] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:42.835049 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:42.835865 20081 raft_consensus.cc:2463] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 0.
I20250116 02:57:42.839025 20122 leader_election.cc:304] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:42.840399 20278 raft_consensus.cc:2798] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:42.840823 20278 raft_consensus.cc:491] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:42.841195 20278 raft_consensus.cc:3054] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:42.842381 20272 tablet_bootstrap.cc:492] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:42.842974 20276 raft_consensus.cc:491] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:42.843490 20276 raft_consensus.cc:513] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.845274 20271 tablet_bootstrap.cc:654] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.845674 20276 leader_election.cc:290] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:42.842375 20270 tablet_bootstrap.cc:492] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:42.849700 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:42.850678 20081 raft_consensus.cc:2463] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 0.
I20250116 02:57:42.850725 20278 raft_consensus.cc:513] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.851689 20272 ts_tablet_manager.cc:1397] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.100s user 0.021s sys 0.020s
I20250116 02:57:42.852480 20196 leader_election.cc:304] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:42.852839 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:42.853514 20230 raft_consensus.cc:2463] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 0.
I20250116 02:57:42.847681 20270 ts_tablet_manager.cc:1397] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.019s
I20250116 02:57:42.853825 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:42.854139 20276 raft_consensus.cc:2798] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:42.854637 20081 raft_consensus.cc:3054] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:42.854890 20276 raft_consensus.cc:491] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:42.855504 20276 raft_consensus.cc:3054] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:42.857543 20270 raft_consensus.cc:357] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.858434 20270 raft_consensus.cc:383] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.858779 20270 raft_consensus.cc:738] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.859539 20270 consensus_queue.cc:260] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.865082 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:42.865945 20290 raft_consensus.cc:491] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:42.866369 20272 raft_consensus.cc:357] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.867878 20272 raft_consensus.cc:383] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.868181 20272 raft_consensus.cc:738] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.865929 20289 raft_consensus.cc:491] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:42.868963 20289 raft_consensus.cc:513] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.867357 20290 raft_consensus.cc:513] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.870318 20276 raft_consensus.cc:513] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.871544 20230 raft_consensus.cc:2388] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 28eb102bb0fe457fbe61295f0e9e967a in current term 1: Already voted for candidate d4721f3cbce642dc82cf320fe38a5c24 in this term.
I20250116 02:57:42.872298 20081 raft_consensus.cc:2463] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 1.
I20250116 02:57:42.872754 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:42.874017 20290 leader_election.cc:290] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:42.874224 20156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:42.874997 20122 leader_election.cc:304] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:42.875313 20156 raft_consensus.cc:2463] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 0.
I20250116 02:57:42.875439 20272 consensus_queue.cc:260] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
W20250116 02:57:42.876087 20122 leader_election.cc:343] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:42.876803 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:42.877697 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:42.878865 20272 ts_tablet_manager.cc:1428] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.026s user 0.003s sys 0.005s
W20250116 02:57:42.879431 20122 leader_election.cc:343] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:42.877753 20155 raft_consensus.cc:2388] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4721f3cbce642dc82cf320fe38a5c24 in current term 1: Already voted for candidate 28eb102bb0fe457fbe61295f0e9e967a in this term.
I20250116 02:57:42.859802 20278 leader_election.cc:290] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:42.881374 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:42.881616 20272 tablet_bootstrap.cc:492] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:42.881708 20195 leader_election.cc:304] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters: ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:42.882061 20290 raft_consensus.cc:2798] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:42.884887 20290 raft_consensus.cc:695] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Becoming Leader. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:42.888582 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:42.886000 20290 consensus_queue.cc:237] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.890233 20080 raft_consensus.cc:2388] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d4721f3cbce642dc82cf320fe38a5c24 in current term 1: Already voted for candidate 28eb102bb0fe457fbe61295f0e9e967a in this term.
I20250116 02:57:42.897719 20277 raft_consensus.cc:491] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:42.897964 20289 leader_election.cc:290] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:42.898416 20277 raft_consensus.cc:513] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
W20250116 02:57:42.900782 20027 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:57:42.902465 20293 raft_consensus.cc:2798] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250116 02:57:42.902812 20119 leader_election.cc:343] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:42.904548 20119 leader_election.cc:304] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a; no voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:42.905146 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:42.905512 20289 raft_consensus.cc:2743] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:42.906582 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:42.907724 20271 tablet_bootstrap.cc:492] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:42.908610 20271 ts_tablet_manager.cc:1397] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.076s user 0.016s sys 0.004s
I20250116 02:57:42.909031 20195 leader_election.cc:304] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24; no voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:42.909582 20270 ts_tablet_manager.cc:1428] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.055s user 0.004s sys 0.010s
I20250116 02:57:42.902989 20293 raft_consensus.cc:491] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:42.910331 20293 raft_consensus.cc:3054] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:42.910807 20270 tablet_bootstrap.cc:492] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:42.913098 20277 leader_election.cc:290] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:42.903828 20276 leader_election.cc:290] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:42.915266 20276 raft_consensus.cc:2743] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:57:42.925055 20270 tablet_bootstrap.cc:654] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.931281 20272 tablet_bootstrap.cc:654] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.931356 19978 catalog_manager.cc:5526] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a reported cstate change: term changed from 0 to 1, leader changed from <none> to 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130). New cstate: current_term: 1 leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:42.933440 20271 raft_consensus.cc:357] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.934303 20271 raft_consensus.cc:383] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.934634 20271 raft_consensus.cc:738] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.936054 20297 raft_consensus.cc:491] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:42.937085 20271 consensus_queue.cc:260] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.936546 20297 raft_consensus.cc:513] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.936939 20293 raft_consensus.cc:513] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.949235 20230 raft_consensus.cc:2388] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 28eb102bb0fe457fbe61295f0e9e967a in current term 1: Already voted for candidate d4721f3cbce642dc82cf320fe38a5c24 in this term.
I20250116 02:57:42.951274 20119 leader_election.cc:304] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a; no voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:42.952245 20290 raft_consensus.cc:3054] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:42.953984 20297 leader_election.cc:290] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:42.955960 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:42.960805 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
W20250116 02:57:42.962003 20195 leader_election.cc:343] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:57:42.962661 20196 leader_election.cc:343] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:42.959753 20156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:42.962911 20270 tablet_bootstrap.cc:492] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:42.963536 20196 leader_election.cc:304] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24; no voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:42.957953 20271 ts_tablet_manager.cc:1428] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.049s user 0.008s sys 0.001s
I20250116 02:57:42.963829 20270 ts_tablet_manager.cc:1397] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.010s
I20250116 02:57:42.963761 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:42.964787 20155 raft_consensus.cc:2463] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 0.
I20250116 02:57:42.964918 20297 raft_consensus.cc:2743] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:42.957165 20293 leader_election.cc:290] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:42.959861 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:42.966243 20046 leader_election.cc:304] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters:
W20250116 02:57:42.968179 20196 leader_election.cc:343] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:42.968732 20271 tablet_bootstrap.cc:492] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:42.968631 20277 raft_consensus.cc:2798] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:42.969187 20277 raft_consensus.cc:491] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:42.969527 20277 raft_consensus.cc:3054] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:42.970089 20270 raft_consensus.cc:357] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.970943 20270 raft_consensus.cc:383] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:42.971251 20270 raft_consensus.cc:738] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:42.972011 20270 consensus_queue.cc:260] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.974083 20290 raft_consensus.cc:2743] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:42.976405 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
W20250116 02:57:42.978412 20044 leader_election.cc:343] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:42.979640 20277 raft_consensus.cc:513] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:42.981400 20271 tablet_bootstrap.cc:654] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:42.981958 20277 leader_election.cc:290] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:42.983966 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:42.983053 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:42.988520 20155 raft_consensus.cc:3054] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:42.982724 20156 raft_consensus.cc:2463] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 1.
I20250116 02:57:42.989111 20270 ts_tablet_manager.cc:1428] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250116 02:57:42.990219 20270 tablet_bootstrap.cc:492] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:42.990645 20195 leader_election.cc:304] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters: ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:42.991580 20293 raft_consensus.cc:2798] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:42.992177 20293 raft_consensus.cc:695] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:42.993048 20293 consensus_queue.cc:237] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
W20250116 02:57:42.988758 20044 leader_election.cc:343] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.000452 20272 tablet_bootstrap.cc:492] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:43.001065 20272 ts_tablet_manager.cc:1397] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.121s user 0.017s sys 0.014s
I20250116 02:57:43.002055 20155 raft_consensus.cc:2463] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 1.
I20250116 02:57:43.003294 20046 leader_election.cc:304] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters: d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:43.004761 20277 raft_consensus.cc:2798] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:43.004592 20272 raft_consensus.cc:357] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.009555 20271 tablet_bootstrap.cc:492] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:43.010141 20271 ts_tablet_manager.cc:1397] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.000s
I20250116 02:57:43.014317 20270 tablet_bootstrap.cc:654] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.015558 20271 raft_consensus.cc:357] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.016530 20271 raft_consensus.cc:383] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.016934 20271 raft_consensus.cc:738] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.006098 20272 raft_consensus.cc:383] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.018051 20277 raft_consensus.cc:695] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:43.018319 20272 raft_consensus.cc:738] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.019433 20277 consensus_queue.cc:237] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.021059 20271 consensus_queue.cc:260] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.019601 19978 catalog_manager.cc:5526] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 1 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:43.024524 20270 tablet_bootstrap.cc:492] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:43.025079 20270 ts_tablet_manager.cc:1397] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.000s
I20250116 02:57:43.019490 20272 consensus_queue.cc:260] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.028390 20271 ts_tablet_manager.cc:1428] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.018s user 0.012s sys 0.000s
I20250116 02:57:43.032848 20272 ts_tablet_manager.cc:1428] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20250116 02:57:43.033638 20271 tablet_bootstrap.cc:492] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:43.033802 20272 tablet_bootstrap.cc:492] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:43.035166 19976 catalog_manager.cc:5526] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 0 to 1, leader changed from <none> to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 1 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:43.041345 20272 tablet_bootstrap.cc:654] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.044112 20270 raft_consensus.cc:357] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.045086 20270 raft_consensus.cc:383] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.045382 20270 raft_consensus.cc:738] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.045580 20271 tablet_bootstrap.cc:654] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.046128 20270 consensus_queue.cc:260] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.048712 20270 ts_tablet_manager.cc:1428] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250116 02:57:43.049715 20270 tablet_bootstrap.cc:492] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:43.052136 20271 tablet_bootstrap.cc:492] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:43.052608 20271 ts_tablet_manager.cc:1397] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20250116 02:57:43.055984 20271 raft_consensus.cc:357] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.056932 20271 raft_consensus.cc:383] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.057246 20271 raft_consensus.cc:738] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.058190 20271 consensus_queue.cc:260] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.060021 20270 tablet_bootstrap.cc:654] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.060843 20271 ts_tablet_manager.cc:1428] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250116 02:57:43.061949 20271 tablet_bootstrap.cc:492] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:43.070119 20271 tablet_bootstrap.cc:654] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.070514 20272 tablet_bootstrap.cc:492] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:43.071152 20272 ts_tablet_manager.cc:1397] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.000s
I20250116 02:57:43.075356 20272 raft_consensus.cc:357] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.076259 20272 raft_consensus.cc:383] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.076444 20270 tablet_bootstrap.cc:492] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:43.076625 20272 raft_consensus.cc:738] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.077044 20270 ts_tablet_manager.cc:1397] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.015s
I20250116 02:57:43.077454 20272 consensus_queue.cc:260] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.080349 20270 raft_consensus.cc:357] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.080344 20272 ts_tablet_manager.cc:1428] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20250116 02:57:43.081395 20270 raft_consensus.cc:383] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.081727 20270 raft_consensus.cc:738] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.082267 20272 tablet_bootstrap.cc:492] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:43.082669 20270 consensus_queue.cc:260] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.083767 20271 tablet_bootstrap.cc:492] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:43.086273 20271 ts_tablet_manager.cc:1397] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.007s
I20250116 02:57:43.088456 20270 ts_tablet_manager.cc:1428] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20250116 02:57:43.089764 20270 tablet_bootstrap.cc:492] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:43.089766 20271 raft_consensus.cc:357] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.090898 20271 raft_consensus.cc:383] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.091253 20271 raft_consensus.cc:738] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.092195 20271 consensus_queue.cc:260] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.095345 20271 ts_tablet_manager.cc:1428] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250116 02:57:43.089872 20272 tablet_bootstrap.cc:654] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.096771 20271 tablet_bootstrap.cc:492] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:43.102782 20270 tablet_bootstrap.cc:654] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.103130 20277 raft_consensus.cc:491] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.103701 20277 raft_consensus.cc:513] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.104974 20271 tablet_bootstrap.cc:654] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.106081 20277 leader_election.cc:290] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.109366 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:43.109411 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.111173 20272 tablet_bootstrap.cc:492] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:43.111683 20272 ts_tablet_manager.cc:1397] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.000s
W20250116 02:57:43.112013 20044 leader_election.cc:343] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.113132 20270 tablet_bootstrap.cc:492] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:43.113654 20270 ts_tablet_manager.cc:1397] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.008s
I20250116 02:57:43.111071 20155 raft_consensus.cc:2463] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 0.
I20250116 02:57:43.115809 20046 leader_election.cc:304] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters: d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:43.116298 20272 raft_consensus.cc:357] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.117201 20277 raft_consensus.cc:2798] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:43.117388 20272 raft_consensus.cc:383] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.116938 20270 raft_consensus.cc:357] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.117648 20277 raft_consensus.cc:491] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:43.117833 20272 raft_consensus.cc:738] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.118098 20270 raft_consensus.cc:383] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.118146 20277 raft_consensus.cc:3054] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.118719 20270 raft_consensus.cc:738] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.119680 20270 consensus_queue.cc:260] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.120157 20290 raft_consensus.cc:491] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.120716 20290 raft_consensus.cc:513] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.123160 20290 leader_election.cc:290] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.125195 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:43.129276 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:43.130236 20230 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 0.
W20250116 02:57:43.127075 20122 leader_election.cc:343] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.134141 20270 ts_tablet_manager.cc:1428] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20250116 02:57:43.135478 20270 tablet_bootstrap.cc:492] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:43.135393 20119 leader_election.cc:304] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters: ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.137176 20290 raft_consensus.cc:2798] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:43.137621 20290 raft_consensus.cc:491] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:43.137992 20290 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.119009 20272 consensus_queue.cc:260] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.143144 20270 tablet_bootstrap.cc:654] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.148891 20272 ts_tablet_manager.cc:1428] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.037s user 0.007s sys 0.001s
I20250116 02:57:43.150368 20272 tablet_bootstrap.cc:492] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:43.151902 20277 raft_consensus.cc:513] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.154582 20277 leader_election.cc:290] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.155606 20270 tablet_bootstrap.cc:492] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:43.156121 20270 ts_tablet_manager.cc:1397] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20250116 02:57:43.155946 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.156196 20271 tablet_bootstrap.cc:492] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:43.156873 20271 ts_tablet_manager.cc:1397] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.043s
W20250116 02:57:43.157438 20044 leader_election.cc:343] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.159184 20270 raft_consensus.cc:357] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.159397 20271 raft_consensus.cc:357] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.159993 20270 raft_consensus.cc:383] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.160189 20271 raft_consensus.cc:383] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.160399 20270 raft_consensus.cc:738] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.159926 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.161010 20155 raft_consensus.cc:3054] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.161291 20270 consensus_queue.cc:260] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.163676 20270 ts_tablet_manager.cc:1428] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250116 02:57:43.160503 20271 raft_consensus.cc:738] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.164844 20271 consensus_queue.cc:260] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.166062 20290 raft_consensus.cc:513] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.167137 20270 tablet_bootstrap.cc:492] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a: Bootstrap starting.
I20250116 02:57:43.169369 20271 ts_tablet_manager.cc:1428] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20250116 02:57:43.170280 20271 tablet_bootstrap.cc:492] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:43.172698 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:43.173358 20270 tablet_bootstrap.cc:654] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a: Neither blocks nor log segments found. Creating new log.
W20250116 02:57:43.174168 20122 leader_election.cc:343] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.174935 20272 tablet_bootstrap.cc:654] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.175486 20271 tablet_bootstrap.cc:654] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.181154 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.181962 20230 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.184353 20155 raft_consensus.cc:2463] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 1.
I20250116 02:57:43.186157 20046 leader_election.cc:304] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters: d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:43.187574 20277 raft_consensus.cc:491] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.188151 20277 raft_consensus.cc:513] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.188802 20305 raft_consensus.cc:2798] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:43.192081 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
W20250116 02:57:43.193972 20046 leader_election.cc:343] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.194767 20277 leader_election.cc:290] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.196455 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
W20250116 02:57:43.198421 20044 leader_election.cc:343] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.198982 20044 leader_election.cc:304] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec07b35db1e64604a8285de3d895b36e; no voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:43.200814 20277 raft_consensus.cc:2743] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:43.206825 20230 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 1.
I20250116 02:57:43.208405 20119 leader_election.cc:304] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters: ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.207954 20290 leader_election.cc:290] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.209502 20289 raft_consensus.cc:2798] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:43.210245 20289 raft_consensus.cc:695] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Becoming Leader. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:43.211298 20289 consensus_queue.cc:237] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.215875 20270 tablet_bootstrap.cc:492] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a: No bootstrap required, opened a new log
I20250116 02:57:43.216557 20270 ts_tablet_manager.cc:1397] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a: Time spent bootstrapping tablet: real 0.050s user 0.014s sys 0.021s
I20250116 02:57:43.220160 20270 raft_consensus.cc:357] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.220901 20305 raft_consensus.cc:695] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:43.221259 20270 raft_consensus.cc:383] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.221710 20270 raft_consensus.cc:738] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.222687 20270 consensus_queue.cc:260] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.222512 20305 consensus_queue.cc:237] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.225304 20272 tablet_bootstrap.cc:492] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:43.225610 20270 ts_tablet_manager.cc:1428] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
I20250116 02:57:43.225981 20272 ts_tablet_manager.cc:1397] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.015s
I20250116 02:57:43.226760 20271 tablet_bootstrap.cc:492] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:43.227226 20271 ts_tablet_manager.cc:1397] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.008s
I20250116 02:57:43.229110 20272 raft_consensus.cc:357] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.230001 20272 raft_consensus.cc:383] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.230417 20272 raft_consensus.cc:738] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.230038 20271 raft_consensus.cc:357] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.230962 20271 raft_consensus.cc:383] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.231415 20271 raft_consensus.cc:738] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.231369 20272 consensus_queue.cc:260] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.232331 20271 consensus_queue.cc:260] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.233764 20272 ts_tablet_manager.cc:1428] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250116 02:57:43.234772 20272 tablet_bootstrap.cc:492] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:43.237051 19976 catalog_manager.cc:5526] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a reported cstate change: term changed from 0 to 1, leader changed from <none> to 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130). New cstate: current_term: 1 leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:43.238826 20293 raft_consensus.cc:491] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.239320 20293 raft_consensus.cc:513] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.240473 20271 ts_tablet_manager.cc:1428] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250116 02:57:43.240828 20272 tablet_bootstrap.cc:654] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.241647 20271 tablet_bootstrap.cc:492] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24: Bootstrap starting.
I20250116 02:57:43.243386 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:43.244119 20080 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 0.
I20250116 02:57:43.245419 20196 leader_election.cc:304] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:43.246318 20297 raft_consensus.cc:2798] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:43.246734 20297 raft_consensus.cc:491] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:43.247007 20297 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.250073 20271 tablet_bootstrap.cc:654] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.253508 20297 raft_consensus.cc:513] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.256119 19979 catalog_manager.cc:5526] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 0 to 1, leader changed from <none> to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 1 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:43.241535 20293 leader_election.cc:290] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:43.257299 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:43.258973 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.265456 20080 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.265654 20155 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 0.
I20250116 02:57:43.272095 20297 leader_election.cc:290] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:43.274652 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.275511 20155 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.275997 20080 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 1.
I20250116 02:57:43.277299 20196 leader_election.cc:304] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:43.278223 20297 raft_consensus.cc:2798] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:43.278848 20297 raft_consensus.cc:695] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:43.280068 20297 consensus_queue.cc:237] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.289073 20271 tablet_bootstrap.cc:492] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24: No bootstrap required, opened a new log
I20250116 02:57:43.289633 20271 ts_tablet_manager.cc:1397] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24: Time spent bootstrapping tablet: real 0.048s user 0.028s sys 0.012s
I20250116 02:57:43.293177 20289 consensus_queue.cc:1035] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.295063 20271 raft_consensus.cc:357] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.295863 20271 raft_consensus.cc:383] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.296144 20271 raft_consensus.cc:738] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.298913 20155 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 1.
I20250116 02:57:43.298682 20290 consensus_queue.cc:1035] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.304765 20290 raft_consensus.cc:491] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.305733 20290 raft_consensus.cc:513] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.303608 20271 consensus_queue.cc:260] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.310128 20271 ts_tablet_manager.cc:1428] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24: Time spent starting tablet: real 0.020s user 0.007s sys 0.001s
I20250116 02:57:43.312558 20272 tablet_bootstrap.cc:492] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:43.307350 19976 catalog_manager.cc:5526] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 1 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:43.315615 20272 ts_tablet_manager.cc:1397] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.081s user 0.030s sys 0.036s
I20250116 02:57:43.318431 20290 leader_election.cc:290] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.319155 20272 raft_consensus.cc:357] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.320022 20272 raft_consensus.cc:383] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.320513 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:43.321309 20229 raft_consensus.cc:2463] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 0.
I20250116 02:57:43.322592 20119 leader_election.cc:304] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:43.320312 20079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:43.323904 20290 raft_consensus.cc:2798] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:43.324388 20290 raft_consensus.cc:491] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:43.324883 20290 raft_consensus.cc:3054] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.320545 20272 raft_consensus.cc:738] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
W20250116 02:57:43.324422 20122 leader_election.cc:343] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.328321 20272 consensus_queue.cc:260] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.330376 20297 raft_consensus.cc:491] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.330965 20297 raft_consensus.cc:513] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.333264 20297 leader_election.cc:290] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441)
I20250116 02:57:43.334242 20272 ts_tablet_manager.cc:1428] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250116 02:57:43.335449 20272 tablet_bootstrap.cc:492] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:43.336305 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.337152 20155 raft_consensus.cc:2463] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 0.
I20250116 02:57:43.338253 20195 leader_election.cc:304] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:43.339190 20293 raft_consensus.cc:2798] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:43.336123 20297 raft_consensus.cc:491] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.339591 20293 raft_consensus.cc:491] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:43.343720 20293 raft_consensus.cc:3054] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.340054 20297 raft_consensus.cc:513] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.359701 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
W20250116 02:57:43.361315 20196 leader_election.cc:343] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.364194 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:43.365506 20081 raft_consensus.cc:2463] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 0.
I20250116 02:57:43.368279 20196 leader_election.cc:304] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:43.368849 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.369279 20276 raft_consensus.cc:2798] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:43.369499 20155 raft_consensus.cc:2463] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 0.
I20250116 02:57:43.369611 20276 raft_consensus.cc:491] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:43.372042 20276 raft_consensus.cc:3054] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.364369 20297 leader_election.cc:290] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:43.380811 20276 raft_consensus.cc:513] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.384768 20290 raft_consensus.cc:513] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.385154 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.388752 20155 raft_consensus.cc:3054] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.385763 20276 leader_election.cc:290] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:43.386471 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:43.392563 20297 raft_consensus.cc:491] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.393170 20080 raft_consensus.cc:3054] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.393137 20297 raft_consensus.cc:513] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.397109 20079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:43.396857 20293 raft_consensus.cc:513] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.397632 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
W20250116 02:57:43.399261 20122 leader_election.cc:343] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.399756 20156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.400478 20156 raft_consensus.cc:2463] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 0.
I20250116 02:57:43.400646 20290 leader_election.cc:290] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.403537 20195 leader_election.cc:304] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:43.404464 20156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.405108 20156 raft_consensus.cc:3054] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.405830 20276 raft_consensus.cc:2798] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:57:43.406785 20276 raft_consensus.cc:491] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:57:43.407130 20276 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.399791 20293 leader_election.cc:290] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441)
I20250116 02:57:43.407914 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:43.401876 20297 leader_election.cc:290] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
W20250116 02:57:43.409487 20196 leader_election.cc:343] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:57:43.410392 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.411114 20230 raft_consensus.cc:3054] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.406060 20079 raft_consensus.cc:2463] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 0.
I20250116 02:57:43.413244 20276 raft_consensus.cc:513] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.415742 20272 tablet_bootstrap.cc:654] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.416739 20079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:43.417403 20079 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.403095 20155 raft_consensus.cc:2463] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 1.
I20250116 02:57:43.420643 20195 leader_election.cc:304] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:43.422837 20297 raft_consensus.cc:2798] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:43.423338 20156 raft_consensus.cc:2463] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 1.
I20250116 02:57:43.424935 20195 leader_election.cc:304] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters: ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.426219 20276 leader_election.cc:290] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:43.427088 20276 raft_consensus.cc:2798] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:43.427671 20276 raft_consensus.cc:695] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:43.428462 20230 raft_consensus.cc:2463] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 1.
I20250116 02:57:43.423492 20297 raft_consensus.cc:695] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:43.434825 20156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.434525 20297 consensus_queue.cc:237] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.435523 20156 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.437453 20080 raft_consensus.cc:2463] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 1.
I20250116 02:57:43.428572 20276 consensus_queue.cc:237] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.442853 20272 tablet_bootstrap.cc:492] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:43.443058 20119 leader_election.cc:304] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters: ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.443514 20272 ts_tablet_manager.cc:1397] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.108s user 0.011s sys 0.010s
I20250116 02:57:43.443554 20156 raft_consensus.cc:2463] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 1.
I20250116 02:57:43.444420 20290 raft_consensus.cc:2798] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:43.445384 20290 raft_consensus.cc:695] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Becoming Leader. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:43.445746 20195 leader_election.cc:304] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:43.446276 20290 consensus_queue.cc:237] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.447742 20272 raft_consensus.cc:357] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.448443 20272 raft_consensus.cc:383] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.448715 20272 raft_consensus.cc:738] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.449309 20272 consensus_queue.cc:260] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.446857 20297 raft_consensus.cc:2798] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:57:43.451460 20272 ts_tablet_manager.cc:1428] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250116 02:57:43.451841 20079 raft_consensus.cc:2463] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 1.
I20250116 02:57:43.452355 20272 tablet_bootstrap.cc:492] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:43.455241 20305 consensus_queue.cc:1035] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:43.456245 20297 raft_consensus.cc:695] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:43.457345 20297 consensus_queue.cc:237] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.458223 20272 tablet_bootstrap.cc:654] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.462831 20305 raft_consensus.cc:491] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.463575 20305 raft_consensus.cc:513] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.467114 20305 leader_election.cc:290] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.468457 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:43.468910 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.469822 20154 raft_consensus.cc:2388] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec07b35db1e64604a8285de3d895b36e in current term 1: Already voted for candidate d4721f3cbce642dc82cf320fe38a5c24 in this term.
I20250116 02:57:43.471731 20046 leader_election.cc:304] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec07b35db1e64604a8285de3d895b36e; no voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:43.474421 20305 raft_consensus.cc:3054] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.477962 19978 catalog_manager.cc:5526] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a reported cstate change: term changed from 0 to 1, leader changed from <none> to 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130). New cstate: current_term: 1 leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:43.491241 20297 raft_consensus.cc:491] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.491896 20297 raft_consensus.cc:513] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.494997 20079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07b35db1e64604a8285de3d895b36e" is_pre_election: true
I20250116 02:57:43.498109 20230 raft_consensus.cc:3054] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.498497 20297 leader_election.cc:290] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:43.498961 20277 raft_consensus.cc:491] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.499505 20277 raft_consensus.cc:513] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.502848 20277 leader_election.cc:290] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.504006 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:43.505035 20229 raft_consensus.cc:2388] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec07b35db1e64604a8285de3d895b36e in current term 1: Already voted for candidate 28eb102bb0fe457fbe61295f0e9e967a in this term.
I20250116 02:57:43.507406 20319 consensus_queue.cc:1035] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.515872 20156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.516882 20156 raft_consensus.cc:2388] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4721f3cbce642dc82cf320fe38a5c24 in current term 1: Already voted for candidate ec07b35db1e64604a8285de3d895b36e in this term.
I20250116 02:57:43.516809 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.518030 20195 leader_election.cc:304] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24; no voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.519418 20297 raft_consensus.cc:3054] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.520380 20305 raft_consensus.cc:2743] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:43.522464 20046 leader_election.cc:304] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec07b35db1e64604a8285de3d895b36e; no voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:43.528478 20277 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.528642 20297 raft_consensus.cc:2743] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:43.530158 20297 consensus_queue.cc:1035] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:43.532575 19976 catalog_manager.cc:5526] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 1 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:43.534338 20272 tablet_bootstrap.cc:492] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:43.535207 20272 ts_tablet_manager.cc:1397] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.083s user 0.028s sys 0.005s
I20250116 02:57:43.534952 19976 catalog_manager.cc:5526] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 1 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:43.538447 20272 raft_consensus.cc:357] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.539541 20272 raft_consensus.cc:383] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.539916 20272 raft_consensus.cc:738] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.539536 20277 raft_consensus.cc:2743] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:43.541074 20272 consensus_queue.cc:260] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.543718 20272 ts_tablet_manager.cc:1428] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250116 02:57:43.544967 20272 tablet_bootstrap.cc:492] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e: Bootstrap starting.
I20250116 02:57:43.552870 20272 tablet_bootstrap.cc:654] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e: Neither blocks nor log segments found. Creating new log.
I20250116 02:57:43.555140 19979 catalog_manager.cc:5526] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 1 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:43.557152 20293 consensus_queue.cc:1035] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.564566 20272 tablet_bootstrap.cc:492] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e: No bootstrap required, opened a new log
I20250116 02:57:43.565073 20272 ts_tablet_manager.cc:1397] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.009s
I20250116 02:57:43.566295 20277 raft_consensus.cc:491] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.566782 20277 raft_consensus.cc:513] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.567519 20272 raft_consensus.cc:357] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.569039 20272 raft_consensus.cc:383] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:57:43.569416 20272 raft_consensus.cc:738] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Initialized, Role: FOLLOWER
I20250116 02:57:43.570259 20272 consensus_queue.cc:260] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.570592 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:43.571182 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.571533 20230 raft_consensus.cc:2388] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec07b35db1e64604a8285de3d895b36e in current term 1: Already voted for candidate 28eb102bb0fe457fbe61295f0e9e967a in this term.
I20250116 02:57:43.572785 20272 ts_tablet_manager.cc:1428] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250116 02:57:43.569085 20277 leader_election.cc:290] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.578998 20046 leader_election.cc:304] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec07b35db1e64604a8285de3d895b36e; no voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:43.579313 20277 raft_consensus.cc:491] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:57:43.579790 20319 raft_consensus.cc:3054] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.579762 20277 raft_consensus.cc:513] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.583238 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24" is_pre_election: true
I20250116 02:57:43.583539 20277 leader_election.cc:290] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.585572 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a" is_pre_election: true
I20250116 02:57:43.586452 20153 raft_consensus.cc:2388] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec07b35db1e64604a8285de3d895b36e in current term 1: Already voted for candidate d4721f3cbce642dc82cf320fe38a5c24 in this term.
I20250116 02:57:43.587553 20046 leader_election.cc:304] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec07b35db1e64604a8285de3d895b36e; no voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:43.588799 20277 raft_consensus.cc:3054] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:57:43.593480 20319 raft_consensus.cc:2743] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:43.595897 20277 raft_consensus.cc:2743] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:57:43.611589 20230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:44170
I20250116 02:57:43.612301 20230 raft_consensus.cc:604] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.627426 20153 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:40906
I20250116 02:57:43.627962 20153 raft_consensus.cc:604] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.633285 20230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:44170
I20250116 02:57:43.633826 20230 raft_consensus.cc:604] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.639431 20153 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827733d443b142479360c6e9dbf5483b"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:40906
I20250116 02:57:43.640043 20153 raft_consensus.cc:604] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.643350 20277 consensus_queue.cc:1035] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:43.646121 20154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:40906
I20250116 02:57:43.646842 20154 raft_consensus.cc:604] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:43.652980 20229 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
mode: GRACEFUL
new_leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:44170
I20250116 02:57:43.653635 20229 raft_consensus.cc:604] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Received request to transfer leadership to TS 28eb102bb0fe457fbe61295f0e9e967a
I20250116 02:57:43.656245 20337 consensus_queue.cc:1035] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.662142 20230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
mode: GRACEFUL
new_leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:44170
I20250116 02:57:43.662950 20230 raft_consensus.cc:604] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Received request to transfer leadership to TS 28eb102bb0fe457fbe61295f0e9e967a
I20250116 02:57:43.676991 20079 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
mode: GRACEFUL
new_leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:36732
I20250116 02:57:43.677592 20079 raft_consensus.cc:604] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Received request to transfer leadership to TS 28eb102bb0fe457fbe61295f0e9e967a
I20250116 02:57:43.682768 20229 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
mode: GRACEFUL
new_leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:44170
I20250116 02:57:43.683531 20229 raft_consensus.cc:604] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Received request to transfer leadership to TS 28eb102bb0fe457fbe61295f0e9e967a
I20250116 02:57:43.688833 20079 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
mode: GRACEFUL
new_leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:36732
I20250116 02:57:43.689438 20079 raft_consensus.cc:604] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Received request to transfer leadership to TS d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:43.692723 20290 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:43.704378 20290 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.715611 20289 raft_consensus.cc:988] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.716199 20290 raft_consensus.cc:1076] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.717962 20079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36718
I20250116 02:57:43.718582 20079 raft_consensus.cc:491] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:43.718959 20079 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.724761 20079 raft_consensus.cc:513] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.727021 20079 leader_election.cc:290] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.727933 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.728003 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.728791 20229 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.728786 20153 raft_consensus.cc:3049] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:43.729310 20153 raft_consensus.cc:738] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:43.730115 20153 consensus_queue.cc:260] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.731395 20153 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.736595 20229 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 2.
I20250116 02:57:43.738322 20044 leader_election.cc:304] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:43.738947 20153 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 2.
I20250116 02:57:43.739133 20277 raft_consensus.cc:2798] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:43.739656 20277 raft_consensus.cc:695] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:43.740550 20277 consensus_queue.cc:237] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.747778 19979 catalog_manager.cc:5526] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 1 to 2, leader changed from 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 2 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:43.760566 20289 raft_consensus.cc:988] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.761057 20290 raft_consensus.cc:1076] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.762745 20079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36718
I20250116 02:57:43.763310 20079 raft_consensus.cc:491] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:43.763618 20079 raft_consensus.cc:3054] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.766752 20293 consensus_queue.cc:1035] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:43.769737 20079 raft_consensus.cc:513] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.772595 20079 leader_election.cc:290] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.772809 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.773506 20154 raft_consensus.cc:3049] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:43.773855 20154 raft_consensus.cc:738] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:43.773976 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.774595 20229 raft_consensus.cc:3054] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.774785 20154 consensus_queue.cc:260] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.776510 20297 raft_consensus.cc:988] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24: : Instructing follower 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:57:43.776976 20297 raft_consensus.cc:1076] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Signalling peer 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:57:43.776196 20154 raft_consensus.cc:3054] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.781065 20156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:40892
I20250116 02:57:43.782719 20156 raft_consensus.cc:491] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:43.783164 20156 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.784579 20297 consensus_queue.cc:1035] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.791265 20229 raft_consensus.cc:2463] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 2.
I20250116 02:57:43.791973 20154 raft_consensus.cc:2463] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 2.
I20250116 02:57:43.792601 20044 leader_election.cc:304] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:43.793469 20277 raft_consensus.cc:2798] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:43.794780 20277 raft_consensus.cc:695] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:43.796087 20156 raft_consensus.cc:513] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.796021 20277 consensus_queue.cc:237] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.803158 20156 leader_election.cc:290] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 2 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.803942 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.804533 20229 raft_consensus.cc:3049] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:43.804805 20229 raft_consensus.cc:738] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:43.805559 20229 consensus_queue.cc:260] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.806783 20229 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.808689 20080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:43.809283 20080 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.812472 19978 catalog_manager.cc:5526] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 1 to 2, leader changed from 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 2 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:43.813390 20229 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 2.
I20250116 02:57:43.814720 20119 leader_election.cc:304] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:43.815496 20290 raft_consensus.cc:2798] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:43.815924 20290 raft_consensus.cc:695] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Becoming Leader. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:43.817104 20080 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 2.
I20250116 02:57:43.816895 20290 consensus_queue.cc:237] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.825901 19979 catalog_manager.cc:5526] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a reported cstate change: term changed from 1 to 2, leader changed from d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131) to 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130). New cstate: current_term: 2 leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:43.850301 20297 consensus_queue.cc:1035] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:43.861387 20297 consensus_queue.cc:1035] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.866806 20297 consensus_queue.cc:1035] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.873359 20293 raft_consensus.cc:988] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24: : Instructing follower 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:57:43.873958 20293 raft_consensus.cc:1076] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Signalling peer 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:57:43.875979 20156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:40892
I20250116 02:57:43.877027 20276 raft_consensus.cc:988] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.877499 20276 raft_consensus.cc:1076] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.882339 20276 consensus_queue.cc:1035] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.883651 20080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36712
I20250116 02:57:43.884395 20080 raft_consensus.cc:491] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:43.884805 20080 raft_consensus.cc:3054] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.876667 20156 raft_consensus.cc:491] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:43.891013 20156 raft_consensus.cc:3054] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.892110 20276 consensus_queue.cc:1035] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:43.893747 20080 raft_consensus.cc:513] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.897943 20156 raft_consensus.cc:513] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.900364 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.901055 20154 raft_consensus.cc:3054] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.901242 20156 leader_election.cc:290] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 2 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.901962 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:43.902601 20081 raft_consensus.cc:3054] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.908807 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744b14b25ab74319abb56a19e21a2d2c" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.909544 20229 raft_consensus.cc:3049] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:43.909884 20229 raft_consensus.cc:738] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:43.910960 20276 raft_consensus.cc:988] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.911392 20276 raft_consensus.cc:1076] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.910856 20229 consensus_queue.cc:260] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.912356 20229 raft_consensus.cc:3054] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.914742 20078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36712
I20250116 02:57:43.920701 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c54e2afff19422a81f41473cb9ae916" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.918582 20081 raft_consensus.cc:2463] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 2.
I20250116 02:57:43.919404 20078 raft_consensus.cc:491] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:43.923806 20078 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.925588 20154 raft_consensus.cc:2463] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 2.
I20250116 02:57:43.926144 20122 leader_election.cc:304] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:43.926535 20276 consensus_queue.cc:1035] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:57:43.927292 20290 raft_consensus.cc:2798] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:43.928184 20290 raft_consensus.cc:695] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Becoming Leader. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:43.929217 20290 consensus_queue.cc:237] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.930832 20230 raft_consensus.cc:3049] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:43.931234 20230 raft_consensus.cc:738] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:43.931288 20046 leader_election.cc:304] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:43.932168 20230 consensus_queue.cc:260] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.932816 20277 raft_consensus.cc:2798] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:43.933351 20277 raft_consensus.cc:695] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:43.933789 20230 raft_consensus.cc:3054] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.934402 20229 raft_consensus.cc:2463] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 2.
I20250116 02:57:43.934224 20277 consensus_queue.cc:237] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.919466 20080 leader_election.cc:290] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.942723 20289 consensus_queue.cc:1035] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:43.942104 19976 catalog_manager.cc:5526] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a reported cstate change: term changed from 1 to 2, leader changed from d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131) to 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130). New cstate: current_term: 2 leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:43.946659 20230 raft_consensus.cc:2463] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 2.
I20250116 02:57:43.948015 19978 catalog_manager.cc:5526] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 1 to 2, leader changed from d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 2 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:43.951337 20078 raft_consensus.cc:513] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.954843 20289 raft_consensus.cc:988] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.955585 20290 raft_consensus.cc:1076] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:43.959297 20081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "827733d443b142479360c6e9dbf5483b"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36718
I20250116 02:57:43.959615 20156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.960055 20081 raft_consensus.cc:491] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:43.960502 20081 raft_consensus.cc:3054] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.967151 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.967922 20230 raft_consensus.cc:3049] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:43.968271 20230 raft_consensus.cc:738] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:43.968813 20081 raft_consensus.cc:513] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.959368 20078 leader_election.cc:290] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.970456 20230 consensus_queue.cc:260] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.973054 20230 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.974220 20081 leader_election.cc:290] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:43.975329 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:43.976559 20154 raft_consensus.cc:3049] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:43.976954 20154 raft_consensus.cc:738] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:43.977242 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:43.977979 20228 raft_consensus.cc:3054] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.977805 20154 consensus_queue.cc:260] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.979130 20154 raft_consensus.cc:3054] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.985062 20228 raft_consensus.cc:2463] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 2.
I20250116 02:57:43.985009 20230 raft_consensus.cc:2463] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 2.
I20250116 02:57:43.986788 20044 leader_election.cc:304] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:43.988142 20277 raft_consensus.cc:2798] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:43.988217 20154 raft_consensus.cc:2463] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 2.
I20250116 02:57:43.988293 20044 leader_election.cc:304] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters: 28eb102bb0fe457fbe61295f0e9e967a
I20250116 02:57:43.988786 20277 raft_consensus.cc:695] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:43.989795 20337 raft_consensus.cc:2798] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:43.989730 20277 consensus_queue.cc:237] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.990389 20337 raft_consensus.cc:695] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:43.991501 20337 consensus_queue.cc:237] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:43.992758 20305 raft_consensus.cc:988] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e: : Instructing follower 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:57:43.993214 20305 raft_consensus.cc:1076] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Signalling peer 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:57:43.995568 20154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:40900
I20250116 02:57:43.996300 20154 raft_consensus.cc:491] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:43.996747 20154 raft_consensus.cc:3054] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:43.999567 19976 catalog_manager.cc:5526] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 1 to 2, leader changed from 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 2 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:44.002372 20154 raft_consensus.cc:513] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:44.004346 20154 leader_election.cc:290] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 2 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:44.005597 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:44.005882 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:44.006381 20081 raft_consensus.cc:3049] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:44.006563 20228 raft_consensus.cc:3054] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:44.006819 20081 raft_consensus.cc:738] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:44.007723 20081 consensus_queue.cc:260] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:44.009243 20081 raft_consensus.cc:3054] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:44.014148 19976 catalog_manager.cc:5526] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 1 to 2, leader changed from d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 2 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:44.015040 20228 raft_consensus.cc:2463] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 2.
I20250116 02:57:44.016481 20119 leader_election.cc:304] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:44.017784 20359 raft_consensus.cc:2798] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:44.018311 20359 raft_consensus.cc:695] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Becoming Leader. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:44.018384 20081 raft_consensus.cc:2463] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 2.
I20250116 02:57:44.019345 20359 consensus_queue.cc:237] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:44.028789 19976 catalog_manager.cc:5526] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a reported cstate change: term changed from 1 to 2, leader changed from ec07b35db1e64604a8285de3d895b36e (127.18.238.129) to 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130). New cstate: current_term: 2 leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:44.080405 20276 raft_consensus.cc:988] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24: : Instructing follower 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:57:44.080893 20276 raft_consensus.cc:1076] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Signalling peer 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:57:44.082470 20154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:40892
I20250116 02:57:44.083019 20154 raft_consensus.cc:491] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:44.083340 20154 raft_consensus.cc:3054] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:44.089152 20154 raft_consensus.cc:513] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:44.090972 20154 leader_election.cc:290] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 2 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:44.091908 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:44.092213 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:44.092566 20081 raft_consensus.cc:3054] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:44.092797 20228 raft_consensus.cc:3049] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:44.093112 20228 raft_consensus.cc:738] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:44.093809 20228 consensus_queue.cc:260] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:44.095120 20228 raft_consensus.cc:3054] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:44.100097 20081 raft_consensus.cc:2463] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 2.
I20250116 02:57:44.100657 20228 raft_consensus.cc:2463] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 2.
I20250116 02:57:44.101415 20122 leader_election.cc:304] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:44.102180 20359 raft_consensus.cc:2798] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:44.102725 20359 raft_consensus.cc:695] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Becoming Leader. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:44.103561 20359 consensus_queue.cc:237] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:44.110124 19976 catalog_manager.cc:5526] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a reported cstate change: term changed from 1 to 2, leader changed from d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131) to 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130). New cstate: current_term: 2 leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:44.171620 20228 raft_consensus.cc:1270] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.173058 20337 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.181994 20154 raft_consensus.cc:1270] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.183986 20319 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.197062 20228 raft_consensus.cc:1270] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.198397 20319 consensus_queue.cc:1035] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:44.206530 20154 raft_consensus.cc:1270] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.208271 20337 consensus_queue.cc:1035] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:44.233896 20319 raft_consensus.cc:988] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e: : Instructing follower d4721f3cbce642dc82cf320fe38a5c24 to start an election
I20250116 02:57:44.234544 20337 raft_consensus.cc:1076] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Signalling peer d4721f3cbce642dc82cf320fe38a5c24 to start an election
I20250116 02:57:44.236261 20228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:44162
I20250116 02:57:44.236883 20228 raft_consensus.cc:491] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:44.237267 20228 raft_consensus.cc:3054] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:44.242794 20228 raft_consensus.cc:513] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:44.244513 20228 leader_election.cc:290] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 2 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:44.245467 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:44.245649 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9804b2c38548ba84e86f7a058a1829" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:44.246331 20081 raft_consensus.cc:3049] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:57:44.246358 20154 raft_consensus.cc:3054] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:44.246872 20081 raft_consensus.cc:738] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:44.247728 20081 consensus_queue.cc:260] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:44.249115 20081 raft_consensus.cc:3054] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:44.254377 20154 raft_consensus.cc:2463] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 2.
I20250116 02:57:44.255828 20195 leader_election.cc:304] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:44.256146 20081 raft_consensus.cc:2463] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 2.
I20250116 02:57:44.256718 20276 raft_consensus.cc:2798] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:57:44.257223 20276 raft_consensus.cc:695] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:44.258080 20276 consensus_queue.cc:237] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:44.264659 19978 catalog_manager.cc:5526] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 1 to 2, leader changed from ec07b35db1e64604a8285de3d895b36e (127.18.238.129) to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 2 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:44.328902 20228 raft_consensus.cc:1270] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.330430 20359 consensus_queue.cc:1035] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.338405 20081 raft_consensus.cc:1270] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.340206 20278 consensus_queue.cc:1035] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:44.386546 20154 raft_consensus.cc:1270] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.387835 20337 consensus_queue.cc:1035] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.395838 20228 raft_consensus.cc:1270] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.397629 20319 consensus_queue.cc:1035] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:44.424731 20228 raft_consensus.cc:1270] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.426074 20359 consensus_queue.cc:1035] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.432893 20081 raft_consensus.cc:1270] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.434741 20278 consensus_queue.cc:1035] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.474927 20228 raft_consensus.cc:1270] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.476042 20319 consensus_queue.cc:1035] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.484181 20154 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:57:44.491652 20154 raft_consensus.cc:1270] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.493355 20319 consensus_queue.cc:1035] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.512817 20228 raft_consensus.cc:1270] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.514047 20278 consensus_queue.cc:1035] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.522890 20081 raft_consensus.cc:1270] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.525382 20289 consensus_queue.cc:1035] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:44.545372 20228 raft_consensus.cc:1270] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.546793 20289 consensus_queue.cc:1035] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.557583 20081 raft_consensus.cc:1270] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.560075 20278 consensus_queue.cc:1035] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:57:44.582815 20154 raft_consensus.cc:1270] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.584206 20319 consensus_queue.cc:1035] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.592320 20228 raft_consensus.cc:1270] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.593915 20337 consensus_queue.cc:1035] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:57:44.778338 20081 raft_consensus.cc:1270] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.779773 20276 consensus_queue.cc:1035] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:44.791714 20154 raft_consensus.cc:1270] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:57:44.793308 20365 consensus_queue.cc:1035] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:57:47.273394 19386 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20250116 02:57:47.274190 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:47.298514 20081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
mode: GRACEFUL
new_leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:36744
I20250116 02:57:47.299207 20081 raft_consensus.cc:604] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Received request to transfer leadership to TS d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:47.302022 20081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
mode: GRACEFUL
new_leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:36744
I20250116 02:57:47.302599 20081 raft_consensus.cc:604] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Received request to transfer leadership to TS d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:57:47.303769 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250116 02:57:47.304239 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250116 02:57:47.597263 20177 debug-util.cc:398] Leaking SignalData structure 0x7b0800108760 after lost signal to thread 19389
W20250116 02:57:47.598492 20177 debug-util.cc:398] Leaking SignalData structure 0x7b0800108e40 after lost signal to thread 20013
I20250116 02:57:47.598726 20372 raft_consensus.cc:988] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e: : Instructing follower d4721f3cbce642dc82cf320fe38a5c24 to start an election
I20250116 02:57:47.599351 20378 raft_consensus.cc:1076] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Signalling peer d4721f3cbce642dc82cf320fe38a5c24 to start an election
W20250116 02:57:47.600517 20177 debug-util.cc:398] Leaking SignalData structure 0x7b08001af860 after lost signal to thread 20180
I20250116 02:57:47.601255 20229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:44162
W20250116 02:57:47.601640 20177 debug-util.cc:398] Leaking SignalData structure 0x7b08001d2b00 after lost signal to thread 20254
I20250116 02:57:47.601995 20229 raft_consensus.cc:491] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:47.602432 20229 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:47.607117 20229 raft_consensus.cc:513] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:47.608780 20229 leader_election.cc:290] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 3 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:47.609745 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:47.610399 20081 raft_consensus.cc:3049] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:47.610819 20081 raft_consensus.cc:738] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:47.611371 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:47.612179 20153 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:47.611689 20081 consensus_queue.cc:260] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:47.615128 20377 raft_consensus.cc:988] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e: : Instructing follower d4721f3cbce642dc82cf320fe38a5c24 to start an election
I20250116 02:57:47.615605 20377 raft_consensus.cc:1076] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Signalling peer d4721f3cbce642dc82cf320fe38a5c24 to start an election
I20250116 02:57:47.616607 20081 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:47.617218 20227 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:44162
I20250116 02:57:47.617762 20227 raft_consensus.cc:491] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:47.618113 20227 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:47.623807 20153 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 3.
I20250116 02:57:47.624481 20081 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 3.
I20250116 02:57:47.625331 20195 leader_election.cc:304] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:47.626284 20380 raft_consensus.cc:2798] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:47.625870 20227 raft_consensus.cc:513] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:47.626847 20380 raft_consensus.cc:695] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:47.627782 20380 consensus_queue.cc:237] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:47.629508 20081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:57:47.630240 20081 raft_consensus.cc:3049] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:47.630447 20227 leader_election.cc:290] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 3 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:57:47.630645 20081 raft_consensus.cc:738] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:47.631711 20081 consensus_queue.cc:260] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:47.632764 20153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:47.633628 20153 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:47.634114 20081 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:47.643033 20153 raft_consensus.cc:2463] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 3.
I20250116 02:57:47.643042 20081 raft_consensus.cc:2463] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 3.
I20250116 02:57:47.644721 20195 leader_election.cc:304] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:57:47.646114 20379 raft_consensus.cc:2798] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:47.654934 20379 raft_consensus.cc:695] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:47.655845 20379 consensus_queue.cc:237] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:47.658967 19979 catalog_manager.cc:5526] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 2 to 3, leader changed from ec07b35db1e64604a8285de3d895b36e (127.18.238.129) to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 3 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:47.679293 19979 catalog_manager.cc:5526] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 2 to 3, leader changed from ec07b35db1e64604a8285de3d895b36e (127.18.238.129) to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 3 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:57:48.100732 20081 raft_consensus.cc:1270] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:48.102007 20379 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:57:48.115088 20153 raft_consensus.cc:1270] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:48.117683 20379 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:48.212719 20081 raft_consensus.cc:1270] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:48.214434 20379 consensus_queue.cc:1035] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:48.233016 20153 raft_consensus.cc:1270] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:48.235227 20379 consensus_queue.cc:1035] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:48.305119 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:48.311350 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:48.311864 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:49.313208 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:49.320638 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:49.321171 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:50.322187 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:50.327800 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:50.328341 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:51.329313 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:51.334935 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:51.335661 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:52.336750 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:52.342993 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:52.343430 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:53.344386 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:53.351842 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:53.352516 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:54.353370 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:54.359261 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:54.359833 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:55.360808 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:55.367480 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:55.367947 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:56.368808 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:57:56.375074 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:57:56.375551 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:57:57.413030 19386 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20250116 02:57:57.421201 20229 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:44170
I20250116 02:57:57.422474 20229 raft_consensus.cc:604] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:57.428712 20229 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:44170
I20250116 02:57:57.429314 20229 raft_consensus.cc:604] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:57.441208 20154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:40906
I20250116 02:57:57.441872 20154 raft_consensus.cc:604] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:57.447666 20154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:40906
I20250116 02:57:57.448236 20154 raft_consensus.cc:604] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:57.453459 20154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
mode: GRACEFUL
new_leader_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:40906
I20250116 02:57:57.454154 20154 raft_consensus.cc:604] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Received request to transfer leadership to TS ec07b35db1e64604a8285de3d895b36e
I20250116 02:57:57.462117 20408 raft_consensus.cc:988] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:57.462720 20400 raft_consensus.cc:1076] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:57.464531 20078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36712
I20250116 02:57:57.465081 20078 raft_consensus.cc:491] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:57.465523 20078 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:57.472685 20078 raft_consensus.cc:513] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.474895 20078 leader_election.cc:290] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 4 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:57.476030 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:57.476145 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388b9f9b579d4f0ab9758daf60f3a680" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:57.476794 20154 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:57.476977 20229 raft_consensus.cc:3049] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:57:57.477442 20229 raft_consensus.cc:738] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:57.478242 20229 consensus_queue.cc:260] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.479437 20229 raft_consensus.cc:3054] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:57.483167 20154 raft_consensus.cc:2463] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 4.
I20250116 02:57:57.484632 20046 leader_election.cc:304] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:57.485685 20396 raft_consensus.cc:2798] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:57:57.485940 20229 raft_consensus.cc:2463] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 4.
I20250116 02:57:57.486829 20396 raft_consensus.cc:695] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 4 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:57.487818 20396 consensus_queue.cc:237] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.495553 19976 catalog_manager.cc:5526] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 3 to 4, leader changed from d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 4 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:57.499711 20408 raft_consensus.cc:988] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:57.500460 20400 raft_consensus.cc:1076] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:57.502250 20078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36712
I20250116 02:57:57.503013 20078 raft_consensus.cc:491] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:57.503464 20078 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:57.510655 20078 raft_consensus.cc:513] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.512712 20078 leader_election.cc:290] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 4 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:57.513710 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:57.513705 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:57.514389 20229 raft_consensus.cc:3049] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:57:57.514735 20154 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:57.514801 20229 raft_consensus.cc:738] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:57:57.515833 20229 consensus_queue.cc:260] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.516924 20229 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:57:57.520330 20154 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 4.
I20250116 02:57:57.521530 20046 leader_election.cc:304] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:57.522447 20396 raft_consensus.cc:2798] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:57:57.522804 20229 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 4.
I20250116 02:57:57.523072 20396 raft_consensus.cc:695] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 4 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:57.524031 20396 consensus_queue.cc:237] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.532336 19976 catalog_manager.cc:5526] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 3 to 4, leader changed from d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 4 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:57.676119 20407 raft_consensus.cc:988] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:57.676734 20416 raft_consensus.cc:1076] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:57.678514 20078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36718
I20250116 02:57:57.679184 20078 raft_consensus.cc:491] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:57.679484 20078 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:57.685791 20078 raft_consensus.cc:513] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.687506 20078 leader_election.cc:290] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 3 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:57.688365 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:57.688762 20407 raft_consensus.cc:988] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:57.689095 20154 raft_consensus.cc:3049] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:57.688501 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:57.689706 20154 raft_consensus.cc:738] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:57.689746 20416 raft_consensus.cc:1076] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:57.690068 20229 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:57.691522 20154 consensus_queue.cc:260] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.692960 20154 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:57.692750 20078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36718
I20250116 02:57:57.697597 20078 raft_consensus.cc:491] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:57.698081 20078 raft_consensus.cc:3054] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:57.700495 20229 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 3.
I20250116 02:57:57.701694 20044 leader_election.cc:304] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:57.701921 20154 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 3.
I20250116 02:57:57.702394 20396 raft_consensus.cc:2798] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:57.702867 20396 raft_consensus.cc:695] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 3 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:57.703801 20396 consensus_queue.cc:237] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.705183 20078 raft_consensus.cc:513] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.707309 20078 leader_election.cc:290] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 3 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:57.708190 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:57.710639 20154 raft_consensus.cc:3049] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:57.712338 20154 raft_consensus.cc:738] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:57.712477 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b150dc2209849c38456b6fb80e4ff53" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:57.713171 20154 consensus_queue.cc:260] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.714211 20229 raft_consensus.cc:3054] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:57.715682 20154 raft_consensus.cc:3054] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:57.718376 19976 catalog_manager.cc:5526] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 2 to 3, leader changed from 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 3 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:57.722821 20229 raft_consensus.cc:2463] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 3.
I20250116 02:57:57.723171 20154 raft_consensus.cc:2463] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 3.
I20250116 02:57:57.724141 20044 leader_election.cc:304] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:57.724956 20396 raft_consensus.cc:2798] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:57.725332 20396 raft_consensus.cc:695] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 3 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:57.726050 20396 consensus_queue.cc:237] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:57.733623 19976 catalog_manager.cc:5526] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 2 to 3, leader changed from 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 3 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:57.871299 20154 raft_consensus.cc:1270] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:57:57.872687 20396 consensus_queue.cc:1035] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:57:57.883817 20229 raft_consensus.cc:1270] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:57:57.885702 20409 consensus_queue.cc:1035] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:57:57.930384 20154 raft_consensus.cc:1270] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:57:57.931583 20409 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:57:57.941179 20229 raft_consensus.cc:1270] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:57:57.943092 20396 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:57:58.007637 20417 raft_consensus.cc:988] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a: : Instructing follower ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:58.008071 20417 raft_consensus.cc:1076] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Signalling peer ec07b35db1e64604a8285de3d895b36e to start an election
I20250116 02:57:58.009588 20078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
from {username='slave'} at 127.0.0.1:36718
I20250116 02:57:58.010061 20078 raft_consensus.cc:491] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:57:58.010325 20078 raft_consensus.cc:3054] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:58.015038 20078 raft_consensus.cc:513] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:58.017752 20078 leader_election.cc:290] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 3 election: Requested vote from peers 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:57:58.018353 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:57:58.018754 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5119b4436ca4ae7b847cb5a33148178" candidate_uuid: "ec07b35db1e64604a8285de3d895b36e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:57:58.019088 20154 raft_consensus.cc:3049] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:57:58.019295 20229 raft_consensus.cc:3054] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:58.019430 20154 raft_consensus.cc:738] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:57:58.020249 20154 consensus_queue.cc:260] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:58.021569 20154 raft_consensus.cc:3054] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:57:58.026965 20229 raft_consensus.cc:2463] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 3.
I20250116 02:57:58.028076 20044 leader_election.cc:304] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:57:58.028730 20396 raft_consensus.cc:2798] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:57:58.029163 20396 raft_consensus.cc:695] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 3 LEADER]: Becoming Leader. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:57:58.029533 20154 raft_consensus.cc:2463] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07b35db1e64604a8285de3d895b36e in term 3.
I20250116 02:57:58.029982 20396 consensus_queue.cc:237] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:57:58.037530 19976 catalog_manager.cc:5526] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e reported cstate change: term changed from 2 to 3, leader changed from 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130) to ec07b35db1e64604a8285de3d895b36e (127.18.238.129). New cstate: current_term: 3 leader_uuid: "ec07b35db1e64604a8285de3d895b36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:57:58.173570 20229 raft_consensus.cc:1270] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:58.174758 20396 consensus_queue.cc:1035] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:57:58.184376 20154 raft_consensus.cc:1270] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:58.186489 20396 consensus_queue.cc:1035] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:57:58.219666 20154 raft_consensus.cc:1270] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:58.221141 20396 consensus_queue.cc:1035] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:57:58.230012 20229 raft_consensus.cc:1270] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:58.231662 20422 consensus_queue.cc:1035] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:57:58.423512 20154 raft_consensus.cc:1270] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:58.424830 20396 consensus_queue.cc:1035] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:57:58.437601 20227 raft_consensus.cc:1270] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Refusing update from remote peer ec07b35db1e64604a8285de3d895b36e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:57:58.439685 20434 consensus_queue.cc:1035] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:00.498584 19386 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20250116 02:58:00.499728 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:00.507196 20078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827733d443b142479360c6e9dbf5483b"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
mode: GRACEFUL
new_leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:36744
I20250116 02:58:00.507907 20078 raft_consensus.cc:604] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Received request to transfer leadership to TS 28eb102bb0fe457fbe61295f0e9e967a
I20250116 02:58:00.510388 20078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
mode: GRACEFUL
new_leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:36744
I20250116 02:58:00.510887 20078 raft_consensus.cc:604] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 4 LEADER]: Received request to transfer leadership to TS d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:58:00.513216 20078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
mode: GRACEFUL
new_leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:36744
I20250116 02:58:00.513651 20078 raft_consensus.cc:604] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 3 LEADER]: Received request to transfer leadership to TS 28eb102bb0fe457fbe61295f0e9e967a
I20250116 02:58:00.515774 20078 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153"
dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
mode: GRACEFUL
new_leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:36744
I20250116 02:58:00.516213 20078 raft_consensus.cc:604] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Received request to transfer leadership to TS d4721f3cbce642dc82cf320fe38a5c24
I20250116 02:58:00.517287 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250116 02:58:00.517910 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:00.670014 20442 raft_consensus.cc:988] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e: : Instructing follower 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:58:00.670571 20439 raft_consensus.cc:1076] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 3 LEADER]: Signalling peer 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:58:00.672206 20154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:40900
I20250116 02:58:00.672701 20154 raft_consensus.cc:491] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:00.673069 20154 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:58:00.678882 20154 raft_consensus.cc:513] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:00.681227 20154 leader_election.cc:290] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 4 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:58:00.682576 20078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:58:00.682916 20227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a755ec1e74245599302dfc6394e26e4" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:58:00.683619 20227 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:58:00.683552 20078 raft_consensus.cc:3049] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:58:00.684118 20442 raft_consensus.cc:988] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e: : Instructing follower d4721f3cbce642dc82cf320fe38a5c24 to start an election
I20250116 02:58:00.684129 20078 raft_consensus.cc:738] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:58:00.684685 20439 raft_consensus.cc:1076] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 4 LEADER]: Signalling peer d4721f3cbce642dc82cf320fe38a5c24 to start an election
I20250116 02:58:00.685298 20078 consensus_queue.cc:260] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:00.686517 20229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:44162
I20250116 02:58:00.686969 20078 raft_consensus.cc:3054] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:58:00.687084 20229 raft_consensus.cc:491] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:00.687451 20229 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:58:00.692245 20227 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 4.
I20250116 02:58:00.693902 20119 leader_election.cc:304] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:58:00.695125 20440 raft_consensus.cc:2798] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:58:00.695628 20078 raft_consensus.cc:2463] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 4.
I20250116 02:58:00.697675 20440 raft_consensus.cc:695] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 4 LEADER]: Becoming Leader. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:58:00.698922 20440 consensus_queue.cc:237] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:00.696202 20229 raft_consensus.cc:513] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:00.703861 20229 leader_election.cc:290] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 5 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:58:00.704779 20078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:58:00.705458 20078 raft_consensus.cc:3049] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:58:00.705688 20078 raft_consensus.cc:738] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:58:00.706368 20078 consensus_queue.cc:260] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:00.706481 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de1db99051645bd97c9e3fb8c65232c" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:58:00.707180 20154 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:58:00.707276 20078 raft_consensus.cc:3054] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:58:00.715260 20154 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 5.
I20250116 02:58:00.715297 20078 raft_consensus.cc:2463] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 5.
I20250116 02:58:00.715472 19976 catalog_manager.cc:5526] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a reported cstate change: term changed from 3 to 4, leader changed from ec07b35db1e64604a8285de3d895b36e (127.18.238.129) to 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130). New cstate: current_term: 4 leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:00.716785 20196 leader_election.cc:304] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4721f3cbce642dc82cf320fe38a5c24, ec07b35db1e64604a8285de3d895b36e; no voters:
I20250116 02:58:00.717710 20441 raft_consensus.cc:2798] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:58:00.719141 20441 raft_consensus.cc:695] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 5 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:58:00.720052 20441 consensus_queue.cc:237] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:00.732724 19978 catalog_manager.cc:5526] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 4 to 5, leader changed from ec07b35db1e64604a8285de3d895b36e (127.18.238.129) to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 5 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:00.909543 20442 raft_consensus.cc:988] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e: : Instructing follower 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:58:00.909938 20442 raft_consensus.cc:1076] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Signalling peer 28eb102bb0fe457fbe61295f0e9e967a to start an election
I20250116 02:58:00.911305 20154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "827733d443b142479360c6e9dbf5483b"
dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
from {username='slave'} at 127.0.0.1:40900
I20250116 02:58:00.911801 20154 raft_consensus.cc:491] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:00.912065 20154 raft_consensus.cc:3054] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:00.916365 20154 raft_consensus.cc:513] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:00.918237 20154 leader_election.cc:290] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 3 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131:33605)
I20250116 02:58:00.919265 20078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:58:00.919354 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827733d443b142479360c6e9dbf5483b" candidate_uuid: "28eb102bb0fe457fbe61295f0e9e967a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
I20250116 02:58:00.920033 20078 raft_consensus.cc:3049] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:58:00.920096 20229 raft_consensus.cc:3054] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:00.920400 20078 raft_consensus.cc:738] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:58:00.921094 20078 consensus_queue.cc:260] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:00.922083 20078 raft_consensus.cc:3054] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:00.925354 20229 raft_consensus.cc:2463] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 3.
I20250116 02:58:00.926653 20119 leader_election.cc:304] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:58:00.927137 20078 raft_consensus.cc:2463] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb102bb0fe457fbe61295f0e9e967a in term 3.
I20250116 02:58:00.927429 20440 raft_consensus.cc:2798] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:58:00.927958 20440 raft_consensus.cc:695] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 3 LEADER]: Becoming Leader. State: Replica: 28eb102bb0fe457fbe61295f0e9e967a, State: Running, Role: LEADER
I20250116 02:58:00.928860 20440 consensus_queue.cc:237] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:00.936900 19978 catalog_manager.cc:5526] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a reported cstate change: term changed from 2 to 3, leader changed from ec07b35db1e64604a8285de3d895b36e (127.18.238.129) to 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130). New cstate: current_term: 3 leader_uuid: "28eb102bb0fe457fbe61295f0e9e967a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:01.090340 20439 raft_consensus.cc:988] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e: : Instructing follower d4721f3cbce642dc82cf320fe38a5c24 to start an election
I20250116 02:58:01.090919 20439 raft_consensus.cc:1076] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Signalling peer d4721f3cbce642dc82cf320fe38a5c24 to start an election
I20250116 02:58:01.092379 20229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153"
dest_uuid: "d4721f3cbce642dc82cf320fe38a5c24"
from {username='slave'} at 127.0.0.1:44162
I20250116 02:58:01.092841 20229 raft_consensus.cc:491] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:01.093063 20229 raft_consensus.cc:3054] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:01.098878 20229 raft_consensus.cc:513] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:01.100551 20229 leader_election.cc:290] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 3 election: Requested vote from peers ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441), 28eb102bb0fe457fbe61295f0e9e967a (127.18.238.130:46575)
I20250116 02:58:01.102066 20078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec07b35db1e64604a8285de3d895b36e"
I20250116 02:58:01.102075 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc97c03c06d4af2b969daa6c856f153" candidate_uuid: "d4721f3cbce642dc82cf320fe38a5c24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "28eb102bb0fe457fbe61295f0e9e967a"
I20250116 02:58:01.103011 20154 raft_consensus.cc:3054] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:01.102926 20078 raft_consensus.cc:3049] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:58:01.103542 20078 raft_consensus.cc:738] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ec07b35db1e64604a8285de3d895b36e, State: Running, Role: LEADER
I20250116 02:58:01.104427 20078 consensus_queue.cc:260] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:01.105894 20078 raft_consensus.cc:3054] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:01.111227 20154 raft_consensus.cc:2463] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 3.
I20250116 02:58:01.112665 20195 leader_election.cc:304] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb102bb0fe457fbe61295f0e9e967a, d4721f3cbce642dc82cf320fe38a5c24; no voters:
I20250116 02:58:01.113570 20078 raft_consensus.cc:2463] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4721f3cbce642dc82cf320fe38a5c24 in term 3.
I20250116 02:58:01.113622 20441 raft_consensus.cc:2798] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:58:01.114204 20441 raft_consensus.cc:695] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Becoming Leader. State: Replica: d4721f3cbce642dc82cf320fe38a5c24, State: Running, Role: LEADER
I20250116 02:58:01.115080 20441 consensus_queue.cc:237] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } }
I20250116 02:58:01.122989 19978 catalog_manager.cc:5526] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 reported cstate change: term changed from 2 to 3, leader changed from ec07b35db1e64604a8285de3d895b36e (127.18.238.129) to d4721f3cbce642dc82cf320fe38a5c24 (127.18.238.131). New cstate: current_term: 3 leader_uuid: "d4721f3cbce642dc82cf320fe38a5c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:01.173434 20078 raft_consensus.cc:1270] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:58:01.174940 20440 consensus_queue.cc:1035] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:01.181753 20229 raft_consensus.cc:1270] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:58:01.183317 20440 consensus_queue.cc:1035] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:01.225514 20078 raft_consensus.cc:1270] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 5 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:58:01.226857 20441 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:01.235857 20154 raft_consensus.cc:1270] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 5 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:58:01.237548 20460 consensus_queue.cc:1035] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:01.404326 20229 raft_consensus.cc:1270] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:01.405838 20443 consensus_queue.cc:1035] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4721f3cbce642dc82cf320fe38a5c24" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:01.418682 20078 raft_consensus.cc:1270] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Refusing update from remote peer 28eb102bb0fe457fbe61295f0e9e967a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:01.420780 20443 consensus_queue.cc:1035] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:01.514659 20154 raft_consensus.cc:1270] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:01.515993 20460 consensus_queue.cc:1035] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb102bb0fe457fbe61295f0e9e967a" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:01.518991 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:01.521306 20078 raft_consensus.cc:1270] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Refusing update from remote peer d4721f3cbce642dc82cf320fe38a5c24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:01.523738 20441 consensus_queue.cc:1035] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07b35db1e64604a8285de3d895b36e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:01.535794 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:58:01.537387 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:02.538668 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:02.545029 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:58:02.545492 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:03.546577 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:03.552299 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:58:03.552857 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:04.553817 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:04.563762 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:58:04.564509 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:05.565599 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:05.572671 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:58:05.573295 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:06.574301 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:06.579994 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:58:06.580543 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:07.581502 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:07.587853 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:58:07.588496 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:08.589406 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:08.595192 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:58:08.595723 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:09.596866 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:09.602885 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250116 02:58:09.603638 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:10.641254 19386 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20250116 02:58:10.641868 19386 tablet_server.cc:178] TabletServer@127.18.238.129:0 shutting down...
W20250116 02:58:10.647572 20122 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20250116 02:58:10.655174 20122 consensus_peers.cc:487] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a -> Peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Couldn't send request to peer ec07b35db1e64604a8285de3d895b36e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:10.675359 19386 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250116 02:58:10.676324 19386 tablet_replica.cc:331] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.677124 19386 raft_consensus.cc:2238] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 2 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.678231 19386 raft_consensus.cc:2267] T 7c54e2afff19422a81f41473cb9ae916 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.680619 19386 tablet_replica.cc:331] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.681306 19386 raft_consensus.cc:2238] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 5 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.681910 19386 raft_consensus.cc:2267] T 8de1db99051645bd97c9e3fb8c65232c P ec07b35db1e64604a8285de3d895b36e [term 5 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.684183 19386 tablet_replica.cc:331] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.684823 19386 raft_consensus.cc:2238] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 4 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.685806 19386 raft_consensus.cc:2267] T 388b9f9b579d4f0ab9758daf60f3a680 P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.688060 19386 tablet_replica.cc:331] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.688611 19386 raft_consensus.cc:2238] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.689162 19386 raft_consensus.cc:2267] T 744b14b25ab74319abb56a19e21a2d2c P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.691216 19386 tablet_replica.cc:331] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.691757 19386 raft_consensus.cc:2238] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 3 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.692690 19386 raft_consensus.cc:2267] T c5119b4436ca4ae7b847cb5a33148178 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.694756 19386 tablet_replica.cc:331] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.695302 19386 raft_consensus.cc:2238] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.695847 19386 raft_consensus.cc:2267] T 9a755ec1e74245599302dfc6394e26e4 P ec07b35db1e64604a8285de3d895b36e [term 4 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.697880 19386 tablet_replica.cc:331] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.698448 19386 raft_consensus.cc:2238] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.698983 19386 raft_consensus.cc:2267] T db9804b2c38548ba84e86f7a058a1829 P ec07b35db1e64604a8285de3d895b36e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.701071 19386 tablet_replica.cc:331] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.701628 19386 raft_consensus.cc:2238] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 3 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.702576 19386 raft_consensus.cc:2267] T 8b150dc2209849c38456b6fb80e4ff53 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.705057 19386 tablet_replica.cc:331] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.705574 19386 raft_consensus.cc:2238] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.706099 19386 raft_consensus.cc:2267] T 827733d443b142479360c6e9dbf5483b P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.708043 19386 tablet_replica.cc:331] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e: stopping tablet replica
I20250116 02:58:10.708534 19386 raft_consensus.cc:2238] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.709015 19386 raft_consensus.cc:2267] T acc97c03c06d4af2b969daa6c856f153 P ec07b35db1e64604a8285de3d895b36e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.743418 19386 tablet_server.cc:195] TabletServer@127.18.238.129:0 shutdown complete.
I20250116 02:58:10.764326 19386 tablet_server.cc:178] TabletServer@127.18.238.130:0 shutting down...
I20250116 02:58:10.794489 19386 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250116 02:58:10.795395 19386 tablet_replica.cc:331] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.796217 19386 raft_consensus.cc:2238] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.796738 19386 raft_consensus.cc:2267] T db9804b2c38548ba84e86f7a058a1829 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.798673 19386 tablet_replica.cc:331] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.799561 19386 raft_consensus.cc:2238] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 4 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.800575 19386 raft_consensus.cc:2267] T 9a755ec1e74245599302dfc6394e26e4 P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.803565 19386 tablet_replica.cc:331] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.804734 19386 raft_consensus.cc:2238] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.805387 19386 raft_consensus.cc:2267] T 8b150dc2209849c38456b6fb80e4ff53 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Raft consensus is shut down!
W20250116 02:58:10.805920 20122 consensus_peers.cc:487] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a -> Peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Couldn't send request to peer ec07b35db1e64604a8285de3d895b36e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:35441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:10.807915 19386 tablet_replica.cc:331] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.808552 19386 raft_consensus.cc:2238] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.809063 19386 raft_consensus.cc:2267] T 388b9f9b579d4f0ab9758daf60f3a680 P 28eb102bb0fe457fbe61295f0e9e967a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.811182 19386 tablet_replica.cc:331] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.811736 19386 raft_consensus.cc:2238] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 5 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.812215 19386 raft_consensus.cc:2267] T 8de1db99051645bd97c9e3fb8c65232c P 28eb102bb0fe457fbe61295f0e9e967a [term 5 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.814244 19386 tablet_replica.cc:331] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.814877 19386 raft_consensus.cc:2238] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.815317 19386 raft_consensus.cc:2267] T 7c54e2afff19422a81f41473cb9ae916 P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.817431 19386 tablet_replica.cc:331] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.817992 19386 raft_consensus.cc:2238] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 2 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.818962 19386 raft_consensus.cc:2267] T 744b14b25ab74319abb56a19e21a2d2c P 28eb102bb0fe457fbe61295f0e9e967a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.821096 19386 tablet_replica.cc:331] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.821621 19386 raft_consensus.cc:2238] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.822069 19386 raft_consensus.cc:2267] T c5119b4436ca4ae7b847cb5a33148178 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.824136 19386 tablet_replica.cc:331] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.824657 19386 raft_consensus.cc:2238] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 3 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.825541 19386 raft_consensus.cc:2267] T 827733d443b142479360c6e9dbf5483b P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.827575 19386 tablet_replica.cc:331] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a: stopping tablet replica
I20250116 02:58:10.828208 19386 raft_consensus.cc:2238] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.828678 19386 raft_consensus.cc:2267] T acc97c03c06d4af2b969daa6c856f153 P 28eb102bb0fe457fbe61295f0e9e967a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.877267 19386 tablet_server.cc:195] TabletServer@127.18.238.130:0 shutdown complete.
W20250116 02:58:10.898581 20196 consensus_peers.cc:487] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 -> Peer ec07b35db1e64604a8285de3d895b36e (127.18.238.129:35441): Couldn't send request to peer ec07b35db1e64604a8285de3d895b36e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:35441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:10.905256 19386 tablet_server.cc:178] TabletServer@127.18.238.131:0 shutting down...
I20250116 02:58:10.941768 19386 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250116 02:58:10.942767 19386 tablet_replica.cc:331] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.943454 19386 raft_consensus.cc:2238] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.944007 19386 raft_consensus.cc:2267] T 8b150dc2209849c38456b6fb80e4ff53 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.946108 19386 tablet_replica.cc:331] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.946837 19386 raft_consensus.cc:2238] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.947782 19386 raft_consensus.cc:2267] T db9804b2c38548ba84e86f7a058a1829 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.949798 19386 tablet_replica.cc:331] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.950384 19386 raft_consensus.cc:2238] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.950870 19386 raft_consensus.cc:2267] T 9a755ec1e74245599302dfc6394e26e4 P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.952870 19386 tablet_replica.cc:331] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.953470 19386 raft_consensus.cc:2238] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.953963 19386 raft_consensus.cc:2267] T 7c54e2afff19422a81f41473cb9ae916 P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.955909 19386 tablet_replica.cc:331] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.956492 19386 raft_consensus.cc:2238] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.956939 19386 raft_consensus.cc:2267] T 388b9f9b579d4f0ab9758daf60f3a680 P d4721f3cbce642dc82cf320fe38a5c24 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.959115 19386 tablet_replica.cc:331] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.959698 19386 raft_consensus.cc:2238] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 5 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.960630 19386 raft_consensus.cc:2267] T 8de1db99051645bd97c9e3fb8c65232c P d4721f3cbce642dc82cf320fe38a5c24 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.962812 19386 tablet_replica.cc:331] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.963362 19386 raft_consensus.cc:2238] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 LEADER]: Raft consensus shutting down.
I20250116 02:58:10.964279 19386 raft_consensus.cc:2267] T acc97c03c06d4af2b969daa6c856f153 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.966470 19386 tablet_replica.cc:331] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.967159 19386 raft_consensus.cc:2238] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.967650 19386 raft_consensus.cc:2267] T 827733d443b142479360c6e9dbf5483b P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.969357 19386 tablet_replica.cc:331] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.969826 19386 raft_consensus.cc:2238] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.970247 19386 raft_consensus.cc:2267] T 744b14b25ab74319abb56a19e21a2d2c P d4721f3cbce642dc82cf320fe38a5c24 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:10.971841 19386 tablet_replica.cc:331] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24: stopping tablet replica
I20250116 02:58:10.972262 19386 raft_consensus.cc:2238] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:10.972671 19386 raft_consensus.cc:2267] T c5119b4436ca4ae7b847cb5a33148178 P d4721f3cbce642dc82cf320fe38a5c24 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:11.004592 19386 tablet_server.cc:195] TabletServer@127.18.238.131:0 shutdown complete.
I20250116 02:58:11.026257 19386 master.cc:537] Master@127.18.238.190:42057 shutting down...
I20250116 02:58:11.049536 19386 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 1 LEADER]: Raft consensus shutting down.
I20250116 02:58:11.050263 19386 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:11.050662 19386 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7f03230ee9634460b5cd609ad1391208: stopping tablet replica
I20250116 02:58:11.071467 19386 master.cc:559] Master@127.18.238.190:42057 shutdown complete.
28eb102bb0fe457fbe61295f0e9e967a 4
d4721f3cbce642dc82cf320fe38a5c24 1
ec07b35db1e64604a8285de3d895b36e 5
28eb102bb0fe457fbe61295f0e9e967a 4
d4721f3cbce642dc82cf320fe38a5c24 3
ec07b35db1e64604a8285de3d895b36e 3
28eb102bb0fe457fbe61295f0e9e967a 1
d4721f3cbce642dc82cf320fe38a5c24 1
ec07b35db1e64604a8285de3d895b36e 8
28eb102bb0fe457fbe61295f0e9e967a 3
d4721f3cbce642dc82cf320fe38a5c24 3
ec07b35db1e64604a8285de3d895b36e 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29383 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250116 02:58:11.120805 19386 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.238.190:37551
I20250116 02:58:11.122579 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:58:11.129794 20481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:58:11.130172 20482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:58:11.131369 19386 server_base.cc:1034] running on GCE node
W20250116 02:58:11.132122 20484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:58:11.133131 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:58:11.133342 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:58:11.133459 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996291133448 us; error 0 us; skew 500 ppm
I20250116 02:58:11.133981 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:58:11.142107 19386 webserver.cc:458] Webserver started at http://127.18.238.190:40325/ using document root <none> and password file <none>
I20250116 02:58:11.142952 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:58:11.143240 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:58:11.143625 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:58:11.145354 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/master-0-root/instance:
uuid: "65b617bfeca24d2cadbd950d2ebf10d8"
format_stamp: "Formatted at 2025-01-16 02:58:11 on dist-test-slave-qn1x"
I20250116 02:58:11.150776 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250116 02:58:11.154860 20489 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:11.155869 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250116 02:58:11.156184 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/master-0-root
uuid: "65b617bfeca24d2cadbd950d2ebf10d8"
format_stamp: "Formatted at 2025-01-16 02:58:11 on dist-test-slave-qn1x"
I20250116 02:58:11.156513 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:58:11.177040 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:58:11.178356 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:58:11.219208 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.190:37551
I20250116 02:58:11.219369 20540 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.190:37551 every 8 connection(s)
I20250116 02:58:11.223665 20541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.235401 20541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8: Bootstrap starting.
I20250116 02:58:11.240914 20541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:11.246040 20541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8: No bootstrap required, opened a new log
I20250116 02:58:11.248610 20541 raft_consensus.cc:357] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b617bfeca24d2cadbd950d2ebf10d8" member_type: VOTER }
I20250116 02:58:11.249226 20541 raft_consensus.cc:383] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:11.249481 20541 raft_consensus.cc:738] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b617bfeca24d2cadbd950d2ebf10d8, State: Initialized, Role: FOLLOWER
I20250116 02:58:11.250177 20541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b617bfeca24d2cadbd950d2ebf10d8" member_type: VOTER }
I20250116 02:58:11.250727 20541 raft_consensus.cc:397] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250116 02:58:11.251003 20541 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250116 02:58:11.251293 20541 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:11.256253 20541 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b617bfeca24d2cadbd950d2ebf10d8" member_type: VOTER }
I20250116 02:58:11.256924 20541 leader_election.cc:304] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65b617bfeca24d2cadbd950d2ebf10d8; no voters:
I20250116 02:58:11.258423 20541 leader_election.cc:290] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250116 02:58:11.258920 20544 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:11.260329 20544 raft_consensus.cc:695] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [term 1 LEADER]: Becoming Leader. State: Replica: 65b617bfeca24d2cadbd950d2ebf10d8, State: Running, Role: LEADER
I20250116 02:58:11.261125 20544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b617bfeca24d2cadbd950d2ebf10d8" member_type: VOTER }
I20250116 02:58:11.261848 20541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [sys.catalog]: configured and running, proceeding with master startup.
I20250116 02:58:11.266583 20546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65b617bfeca24d2cadbd950d2ebf10d8. Latest consensus state: current_term: 1 leader_uuid: "65b617bfeca24d2cadbd950d2ebf10d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b617bfeca24d2cadbd950d2ebf10d8" member_type: VOTER } }
I20250116 02:58:11.267470 20546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [sys.catalog]: This master's current role is: LEADER
I20250116 02:58:11.268419 20545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65b617bfeca24d2cadbd950d2ebf10d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b617bfeca24d2cadbd950d2ebf10d8" member_type: VOTER } }
I20250116 02:58:11.269080 20545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8 [sys.catalog]: This master's current role is: LEADER
I20250116 02:58:11.269665 20551 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250116 02:58:11.274910 20551 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250116 02:58:11.278493 19386 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250116 02:58:11.285748 20551 catalog_manager.cc:1348] Generated new cluster ID: c4f7cad329ee47ceb743bccab2d14411
I20250116 02:58:11.286056 20551 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250116 02:58:11.307075 20551 catalog_manager.cc:1371] Generated new certificate authority record
I20250116 02:58:11.308507 20551 catalog_manager.cc:1505] Loading token signing keys...
I20250116 02:58:11.325708 20551 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8: Generated new TSK 0
I20250116 02:58:11.326433 20551 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250116 02:58:11.345706 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:58:11.352588 20562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:58:11.353935 20563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:58:11.355847 20565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:58:11.356170 19386 server_base.cc:1034] running on GCE node
I20250116 02:58:11.357196 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:58:11.357399 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:58:11.357545 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996291357527 us; error 0 us; skew 500 ppm
I20250116 02:58:11.358255 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:58:11.361353 19386 webserver.cc:458] Webserver started at http://127.18.238.129:45399/ using document root <none> and password file <none>
I20250116 02:58:11.361888 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:58:11.362069 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:58:11.362321 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:58:11.363644 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/instance:
uuid: "f65e8d7f76144fab9725ca32a6328c3f"
format_stamp: "Formatted at 2025-01-16 02:58:11 on dist-test-slave-qn1x"
I20250116 02:58:11.369458 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20250116 02:58:11.373840 20570 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:11.374787 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250116 02:58:11.375161 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root
uuid: "f65e8d7f76144fab9725ca32a6328c3f"
format_stamp: "Formatted at 2025-01-16 02:58:11 on dist-test-slave-qn1x"
I20250116 02:58:11.375478 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:58:11.394845 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:58:11.396337 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:58:11.398128 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:58:11.400841 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:58:11.401095 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:11.401342 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:58:11.401499 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:11.449074 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.129:45195
I20250116 02:58:11.449247 20632 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.129:45195 every 8 connection(s)
I20250116 02:58:11.454147 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:58:11.462165 20637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:58:11.462703 20638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:58:11.466038 20640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:58:11.466513 19386 server_base.cc:1034] running on GCE node
I20250116 02:58:11.468639 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:58:11.468856 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:58:11.469019 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996291469000 us; error 0 us; skew 500 ppm
I20250116 02:58:11.469662 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:58:11.469696 20633 heartbeater.cc:346] Connected to a master server at 127.18.238.190:37551
I20250116 02:58:11.470110 20633 heartbeater.cc:463] Registering TS with master...
I20250116 02:58:11.470935 20633 heartbeater.cc:509] Master 127.18.238.190:37551 requested a full tablet report, sending...
I20250116 02:58:11.472904 19386 webserver.cc:458] Webserver started at http://127.18.238.130:46517/ using document root <none> and password file <none>
I20250116 02:58:11.473456 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:58:11.473659 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:58:11.473712 20506 ts_manager.cc:194] Registered new tserver with Master: f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:11.473949 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:58:11.475484 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/instance:
uuid: "231d91433a404468a44ea0a107345aeb"
format_stamp: "Formatted at 2025-01-16 02:58:11 on dist-test-slave-qn1x"
I20250116 02:58:11.475833 20506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41956
I20250116 02:58:11.480926 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250116 02:58:11.485040 20645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:11.485960 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250116 02:58:11.486265 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root
uuid: "231d91433a404468a44ea0a107345aeb"
format_stamp: "Formatted at 2025-01-16 02:58:11 on dist-test-slave-qn1x"
I20250116 02:58:11.486546 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:58:11.496868 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:58:11.498291 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:58:11.500039 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:58:11.502715 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:58:11.502926 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:11.503218 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:58:11.503371 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:11.550717 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.130:45223
I20250116 02:58:11.550870 20707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.130:45223 every 8 connection(s)
I20250116 02:58:11.555970 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:58:11.564234 20711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:58:11.567188 20712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:58:11.568094 20708 heartbeater.cc:346] Connected to a master server at 127.18.238.190:37551
I20250116 02:58:11.568630 20708 heartbeater.cc:463] Registering TS with master...
W20250116 02:58:11.568737 20714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:58:11.569579 20708 heartbeater.cc:509] Master 127.18.238.190:37551 requested a full tablet report, sending...
I20250116 02:58:11.570264 19386 server_base.cc:1034] running on GCE node
I20250116 02:58:11.571434 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:58:11.571659 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:58:11.571821 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996291571805 us; error 0 us; skew 500 ppm
I20250116 02:58:11.572037 20506 ts_manager.cc:194] Registered new tserver with Master: 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:11.572402 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:58:11.573621 20506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41972
I20250116 02:58:11.575575 19386 webserver.cc:458] Webserver started at http://127.18.238.131:39755/ using document root <none> and password file <none>
I20250116 02:58:11.576299 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:58:11.576539 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:58:11.576768 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:58:11.578320 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/instance:
uuid: "17446c8dacf04461acb843547665250f"
format_stamp: "Formatted at 2025-01-16 02:58:11 on dist-test-slave-qn1x"
I20250116 02:58:11.583565 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20250116 02:58:11.587481 20719 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:11.588740 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250116 02:58:11.589107 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root
uuid: "17446c8dacf04461acb843547665250f"
format_stamp: "Formatted at 2025-01-16 02:58:11 on dist-test-slave-qn1x"
I20250116 02:58:11.589380 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:58:11.612994 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:58:11.614187 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:58:11.615753 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:58:11.618127 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:58:11.618314 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:11.618597 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:58:11.618829 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250116 02:58:11.668344 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.131:43647
I20250116 02:58:11.668479 20781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.131:43647 every 8 connection(s)
I20250116 02:58:11.684492 20782 heartbeater.cc:346] Connected to a master server at 127.18.238.190:37551
I20250116 02:58:11.684943 20782 heartbeater.cc:463] Registering TS with master...
I20250116 02:58:11.685963 20782 heartbeater.cc:509] Master 127.18.238.190:37551 requested a full tablet report, sending...
I20250116 02:58:11.688465 20506 ts_manager.cc:194] Registered new tserver with Master: 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:11.689235 19386 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017404305s
I20250116 02:58:11.690057 20506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41978
I20250116 02:58:11.691965 19386 test_util.cc:274] Using random seed: -151960158
I20250116 02:58:11.719149 20506 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41984:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250116 02:58:11.725879 20506 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250116 02:58:11.943717 20673 tablet_service.cc:1467] Processing CreateTablet for tablet 568ee33beba043a39928aff5f3ae0b3a (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250116 02:58:11.945389 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568ee33beba043a39928aff5f3ae0b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.947381 20669 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8680f966644b519da3afccd8095ab2 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250116 02:58:11.947840 20672 tablet_service.cc:1467] Processing CreateTablet for tablet 6af80665fed7453e80094933a2b5bc2d (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250116 02:58:11.949080 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8680f966644b519da3afccd8095ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.949751 20671 tablet_service.cc:1467] Processing CreateTablet for tablet 54d83a73e060487a8cb3bf72207828c2 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250116 02:58:11.952863 20670 tablet_service.cc:1467] Processing CreateTablet for tablet bf7916e71bc94e1eba4ab8fd75257e48 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250116 02:58:11.954002 20664 tablet_service.cc:1467] Processing CreateTablet for tablet 49cb7aba83e046129200b93f32eb3585 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250116 02:58:11.954502 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d83a73e060487a8cb3bf72207828c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.956028 20665 tablet_service.cc:1467] Processing CreateTablet for tablet b5ac0297b7614c4a8601eded83fff0a2 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250116 02:58:11.952853 20668 tablet_service.cc:1467] Processing CreateTablet for tablet 64313b212d37469f98c8d200fb54ca85 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250116 02:58:11.958664 20667 tablet_service.cc:1467] Processing CreateTablet for tablet 461ef3aac67e4aa49204ce07ff551513 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250116 02:58:11.960274 20666 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1b1a15f38e4ebe9a2943597f827019 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250116 02:58:11.962193 20664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cb7aba83e046129200b93f32eb3585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.962973 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6af80665fed7453e80094933a2b5bc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.965600 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7916e71bc94e1eba4ab8fd75257e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.975199 20665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ac0297b7614c4a8601eded83fff0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.976879 20598 tablet_service.cc:1467] Processing CreateTablet for tablet 568ee33beba043a39928aff5f3ae0b3a (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250116 02:58:11.979337 20598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568ee33beba043a39928aff5f3ae0b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.982918 20668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64313b212d37469f98c8d200fb54ca85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.983561 20597 tablet_service.cc:1467] Processing CreateTablet for tablet 6af80665fed7453e80094933a2b5bc2d (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250116 02:58:11.986953 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 461ef3aac67e4aa49204ce07ff551513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.987377 20597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6af80665fed7453e80094933a2b5bc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.988402 20596 tablet_service.cc:1467] Processing CreateTablet for tablet 54d83a73e060487a8cb3bf72207828c2 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250116 02:58:11.989938 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d83a73e060487a8cb3bf72207828c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.991009 20666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1b1a15f38e4ebe9a2943597f827019. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.992230 20747 tablet_service.cc:1467] Processing CreateTablet for tablet 568ee33beba043a39928aff5f3ae0b3a (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250116 02:58:11.992446 20746 tablet_service.cc:1467] Processing CreateTablet for tablet 6af80665fed7453e80094933a2b5bc2d (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250116 02:58:11.993803 20747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568ee33beba043a39928aff5f3ae0b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:11.994395 20745 tablet_service.cc:1467] Processing CreateTablet for tablet 54d83a73e060487a8cb3bf72207828c2 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250116 02:58:11.995406 20739 tablet_service.cc:1467] Processing CreateTablet for tablet b5ac0297b7614c4a8601eded83fff0a2 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250116 02:58:11.995673 20742 tablet_service.cc:1467] Processing CreateTablet for tablet 64313b212d37469f98c8d200fb54ca85 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250116 02:58:11.997571 20744 tablet_service.cc:1467] Processing CreateTablet for tablet bf7916e71bc94e1eba4ab8fd75257e48 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250116 02:58:12.000044 20743 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8680f966644b519da3afccd8095ab2 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250116 02:58:12.002385 20741 tablet_service.cc:1467] Processing CreateTablet for tablet 461ef3aac67e4aa49204ce07ff551513 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250116 02:58:12.004618 20740 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1b1a15f38e4ebe9a2943597f827019 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250116 02:58:12.007927 20742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64313b212d37469f98c8d200fb54ca85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.006863 20738 tablet_service.cc:1467] Processing CreateTablet for tablet 49cb7aba83e046129200b93f32eb3585 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250116 02:58:12.008670 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d83a73e060487a8cb3bf72207828c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.009302 20739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ac0297b7614c4a8601eded83fff0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.010251 20740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1b1a15f38e4ebe9a2943597f827019. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.012013 20746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6af80665fed7453e80094933a2b5bc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.013207 20738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cb7aba83e046129200b93f32eb3585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.016096 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8680f966644b519da3afccd8095ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.017765 20741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 461ef3aac67e4aa49204ce07ff551513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.023275 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7916e71bc94e1eba4ab8fd75257e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.031797 20595 tablet_service.cc:1467] Processing CreateTablet for tablet bf7916e71bc94e1eba4ab8fd75257e48 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250116 02:58:12.033494 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7916e71bc94e1eba4ab8fd75257e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.040699 20594 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8680f966644b519da3afccd8095ab2 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250116 02:58:12.042469 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8680f966644b519da3afccd8095ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.052145 20593 tablet_service.cc:1467] Processing CreateTablet for tablet 64313b212d37469f98c8d200fb54ca85 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250116 02:58:12.053918 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64313b212d37469f98c8d200fb54ca85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.084261 20671 tablet_service.cc:1467] Processing CreateTablet for tablet c881ef7597c74adfbc9e875bfb6e5f7f (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250116 02:58:12.085870 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c881ef7597c74adfbc9e875bfb6e5f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.097436 20666 tablet_service.cc:1467] Processing CreateTablet for tablet a3ccbf96c6c547e19daa8f5424fa43ee (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250116 02:58:12.099515 20666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ccbf96c6c547e19daa8f5424fa43ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.111479 20592 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1b1a15f38e4ebe9a2943597f827019 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250116 02:58:12.112924 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1b1a15f38e4ebe9a2943597f827019. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.115800 20591 tablet_service.cc:1467] Processing CreateTablet for tablet b5ac0297b7614c4a8601eded83fff0a2 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250116 02:58:12.117111 20591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ac0297b7614c4a8601eded83fff0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.120157 20590 tablet_service.cc:1467] Processing CreateTablet for tablet 49cb7aba83e046129200b93f32eb3585 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250116 02:58:12.121737 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cb7aba83e046129200b93f32eb3585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.124071 20589 tablet_service.cc:1467] Processing CreateTablet for tablet c881ef7597c74adfbc9e875bfb6e5f7f (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250116 02:58:12.125694 20589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c881ef7597c74adfbc9e875bfb6e5f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.144045 20743 tablet_service.cc:1467] Processing CreateTablet for tablet c881ef7597c74adfbc9e875bfb6e5f7f (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250116 02:58:12.154771 20597 tablet_service.cc:1467] Processing CreateTablet for tablet f69213cf66934f1a9aa19c9defd3a4d8 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250116 02:58:12.153149 20797 tablet_bootstrap.cc:492] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:12.156428 20597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69213cf66934f1a9aa19c9defd3a4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.155365 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c881ef7597c74adfbc9e875bfb6e5f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.165071 20669 tablet_service.cc:1467] Processing CreateTablet for tablet 2d25e9e5e0954623a098bf759e01120b (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250116 02:58:12.161350 20595 tablet_service.cc:1467] Processing CreateTablet for tablet 461ef3aac67e4aa49204ce07ff551513 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250116 02:58:12.172024 20798 tablet_bootstrap.cc:492] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:12.173213 20667 tablet_service.cc:1467] Processing CreateTablet for tablet 852e609a81a14d53bed4660b6e8b47f7 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250116 02:58:12.169131 20670 tablet_service.cc:1467] Processing CreateTablet for tablet f69213cf66934f1a9aa19c9defd3a4d8 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250116 02:58:12.182246 20799 tablet_bootstrap.cc:492] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:12.182099 20745 tablet_service.cc:1467] Processing CreateTablet for tablet f69213cf66934f1a9aa19c9defd3a4d8 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250116 02:58:12.184017 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69213cf66934f1a9aa19c9defd3a4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.207017 20665 tablet_service.cc:1467] Processing CreateTablet for tablet 8d16d080f4534834ab906841adf87338 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250116 02:58:12.173468 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 461ef3aac67e4aa49204ce07ff551513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.205920 20673 tablet_service.cc:1467] Processing CreateTablet for tablet 49f9e38171174b70b4373b56dbcd90be (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
W20250116 02:58:12.212980 20490 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 49031.7us
W20250116 02:58:12.208812 20491 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 45494.8us
I20250116 02:58:12.218856 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d25e9e5e0954623a098bf759e01120b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.219720 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69213cf66934f1a9aa19c9defd3a4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.222111 20665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d16d080f4534834ab906841adf87338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.226212 20797 tablet_bootstrap.cc:654] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:12.227317 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f9e38171174b70b4373b56dbcd90be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.230046 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 852e609a81a14d53bed4660b6e8b47f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.235733 20745 tablet_service.cc:1467] Processing CreateTablet for tablet a3ccbf96c6c547e19daa8f5424fa43ee (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250116 02:58:12.237212 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ccbf96c6c547e19daa8f5424fa43ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.237286 20746 tablet_service.cc:1467] Processing CreateTablet for tablet 2d25e9e5e0954623a098bf759e01120b (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250116 02:58:12.239655 20666 tablet_service.cc:1467] Processing CreateTablet for tablet f210b702a7d64ff9970d02aa1f6b4e24 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250116 02:58:12.241752 20746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d25e9e5e0954623a098bf759e01120b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.241719 20671 tablet_service.cc:1467] Processing CreateTablet for tablet eb4a7093fd744af792f09b82b0946a8e (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250116 02:58:12.245366 20666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f210b702a7d64ff9970d02aa1f6b4e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.258565 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb4a7093fd744af792f09b82b0946a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.260910 20593 tablet_service.cc:1467] Processing CreateTablet for tablet a3ccbf96c6c547e19daa8f5424fa43ee (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250116 02:58:12.261209 20740 tablet_service.cc:1467] Processing CreateTablet for tablet 852e609a81a14d53bed4660b6e8b47f7 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250116 02:58:12.262367 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ccbf96c6c547e19daa8f5424fa43ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.267748 20747 tablet_service.cc:1467] Processing CreateTablet for tablet 8d16d080f4534834ab906841adf87338 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250116 02:58:12.269721 20744 tablet_service.cc:1467] Processing CreateTablet for tablet 49f9e38171174b70b4373b56dbcd90be (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250116 02:58:12.273836 20741 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7658a3a42c44f485085e75a4bd7b88 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
W20250116 02:58:12.273975 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:12.262555 20740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 852e609a81a14d53bed4660b6e8b47f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.288511 20741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7658a3a42c44f485085e75a4bd7b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.292584 20668 tablet_service.cc:1467] Processing CreateTablet for tablet 27fcf38273074747a64f8e0c6d42de37 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250116 02:58:12.293777 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f9e38171174b70b4373b56dbcd90be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.273548 20742 tablet_service.cc:1467] Processing CreateTablet for tablet f210b702a7d64ff9970d02aa1f6b4e24 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250116 02:58:12.303725 20742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f210b702a7d64ff9970d02aa1f6b4e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.296159 20799 tablet_bootstrap.cc:654] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:12.284369 20797 tablet_bootstrap.cc:492] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:12.306279 20797 ts_tablet_manager.cc:1397] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.154s user 0.008s sys 0.015s
I20250116 02:58:12.274785 20738 tablet_service.cc:1467] Processing CreateTablet for tablet eb4a7093fd744af792f09b82b0946a8e (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250116 02:58:12.310009 20738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb4a7093fd744af792f09b82b0946a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.310066 20797 raft_consensus.cc:357] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.311318 20797 raft_consensus.cc:383] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:12.311702 20797 raft_consensus.cc:738] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:12.318336 20798 tablet_bootstrap.cc:654] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:12.304980 20743 tablet_service.cc:1467] Processing CreateTablet for tablet 31db17a68ff141f3bd799ccc59c81fcc (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250116 02:58:12.326897 20665 tablet_service.cc:1467] Processing CreateTablet for tablet 31db17a68ff141f3bd799ccc59c81fcc (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250116 02:58:12.351927 20664 tablet_service.cc:1467] Processing CreateTablet for tablet da8115c09ba24b95a9969edcc6ca4d8c (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250116 02:58:12.326817 20797 consensus_queue.cc:260] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.329603 20745 tablet_service.cc:1467] Processing CreateTablet for tablet da8115c09ba24b95a9969edcc6ca4d8c (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250116 02:58:12.334926 20747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d16d080f4534834ab906841adf87338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.354826 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31db17a68ff141f3bd799ccc59c81fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.335880 20598 tablet_service.cc:1467] Processing CreateTablet for tablet 852e609a81a14d53bed4660b6e8b47f7 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250116 02:58:12.356624 20598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 852e609a81a14d53bed4660b6e8b47f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.326244 20595 tablet_service.cc:1467] Processing CreateTablet for tablet 2d25e9e5e0954623a098bf759e01120b (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250116 02:58:12.355576 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8115c09ba24b95a9969edcc6ca4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.342876 20668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27fcf38273074747a64f8e0c6d42de37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.362557 20591 tablet_service.cc:1467] Processing CreateTablet for tablet 8d16d080f4534834ab906841adf87338 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250116 02:58:12.364173 20591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d16d080f4534834ab906841adf87338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.364672 20799 tablet_bootstrap.cc:492] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:12.365224 20799 ts_tablet_manager.cc:1397] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.184s user 0.008s sys 0.005s
I20250116 02:58:12.366981 20665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31db17a68ff141f3bd799ccc59c81fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.368014 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d25e9e5e0954623a098bf759e01120b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.370908 20633 heartbeater.cc:501] Master 127.18.238.190:37551 was elected leader, sending a full tablet report...
I20250116 02:58:12.373968 20797 ts_tablet_manager.cc:1428] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.067s user 0.007s sys 0.000s
I20250116 02:58:12.375144 20797 tablet_bootstrap.cc:492] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:12.378641 20742 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed2f7240f574d40b93ad33069e1a632 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250116 02:58:12.379509 20739 tablet_service.cc:1467] Processing CreateTablet for tablet 27fcf38273074747a64f8e0c6d42de37 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250116 02:58:12.381009 20799 raft_consensus.cc:357] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.382820 20799 raft_consensus.cc:383] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:12.383164 20799 raft_consensus.cc:738] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:12.385267 20597 tablet_service.cc:1467] Processing CreateTablet for tablet f210b702a7d64ff9970d02aa1f6b4e24 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250116 02:58:12.385633 20590 tablet_service.cc:1467] Processing CreateTablet for tablet 49f9e38171174b70b4373b56dbcd90be (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250116 02:58:12.376519 20744 tablet_service.cc:1467] Processing CreateTablet for tablet d92adda8583d41508be4e5a8002cac4d (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250116 02:58:12.396168 20671 tablet_service.cc:1467] Processing CreateTablet for tablet d92adda8583d41508be4e5a8002cac4d (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250116 02:58:12.400146 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92adda8583d41508be4e5a8002cac4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.380019 20742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed2f7240f574d40b93ad33069e1a632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.403496 20798 tablet_bootstrap.cc:492] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:12.408445 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92adda8583d41508be4e5a8002cac4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.411525 20596 tablet_service.cc:1467] Processing CreateTablet for tablet 27fcf38273074747a64f8e0c6d42de37 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250116 02:58:12.413683 20672 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7658a3a42c44f485085e75a4bd7b88 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250116 02:58:12.416496 20664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8115c09ba24b95a9969edcc6ca4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.396476 20594 tablet_service.cc:1467] Processing CreateTablet for tablet eb4a7093fd744af792f09b82b0946a8e (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250116 02:58:12.414753 20666 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed2f7240f574d40b93ad33069e1a632 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250116 02:58:12.419399 20666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed2f7240f574d40b93ad33069e1a632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.419034 20799 consensus_queue.cc:260] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.415684 20667 tablet_service.cc:1467] Processing CreateTablet for tablet b0d2558ee3674a5889434a45c66ac3c1 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250116 02:58:12.420852 20798 ts_tablet_manager.cc:1397] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.249s user 0.014s sys 0.000s
I20250116 02:58:12.422222 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d2558ee3674a5889434a45c66ac3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.386582 20597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f210b702a7d64ff9970d02aa1f6b4e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.430550 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f9e38171174b70b4373b56dbcd90be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.431717 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb4a7093fd744af792f09b82b0946a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.432555 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27fcf38273074747a64f8e0c6d42de37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.441710 20593 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed2f7240f574d40b93ad33069e1a632 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250116 02:58:12.442384 20673 tablet_service.cc:1467] Processing CreateTablet for tablet eb87db38c5904a70bb4b4b6d11afa1ca (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250116 02:58:12.445330 20670 tablet_service.cc:1467] Processing CreateTablet for tablet 0b57de7b372d4101b7f88debce8eea57 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250116 02:58:12.447103 20797 tablet_bootstrap.cc:654] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:12.454232 20738 tablet_service.cc:1467] Processing CreateTablet for tablet eb87db38c5904a70bb4b4b6d11afa1ca (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250116 02:58:12.467362 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed2f7240f574d40b93ad33069e1a632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.452884 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7658a3a42c44f485085e75a4bd7b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.457389 20739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27fcf38273074747a64f8e0c6d42de37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.491437 20803 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:12.498178 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b57de7b372d4101b7f88debce8eea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.499059 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb87db38c5904a70bb4b4b6d11afa1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.499208 20738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb87db38c5904a70bb4b4b6d11afa1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.498210 20803 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.500245 20782 heartbeater.cc:501] Master 127.18.238.190:37551 was elected leader, sending a full tablet report...
I20250116 02:58:12.521008 20598 tablet_service.cc:1467] Processing CreateTablet for tablet 31db17a68ff141f3bd799ccc59c81fcc (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250116 02:58:12.526539 20745 tablet_service.cc:1467] Processing CreateTablet for tablet 99adad9dddaa4121a293aef6fccd1694 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250116 02:58:12.527753 20803 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:12.528352 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99adad9dddaa4121a293aef6fccd1694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.530820 20665 tablet_service.cc:1467] Processing CreateTablet for tablet bddd1f65156348fcb619edd31a8971f8 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250116 02:58:12.532183 20665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddd1f65156348fcb619edd31a8971f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.511804 20746 tablet_service.cc:1467] Processing CreateTablet for tablet b0d2558ee3674a5889434a45c66ac3c1 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250116 02:58:12.540299 20746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d2558ee3674a5889434a45c66ac3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.544947 20595 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7658a3a42c44f485085e75a4bd7b88 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250116 02:58:12.549626 20589 tablet_service.cc:1467] Processing CreateTablet for tablet da8115c09ba24b95a9969edcc6ca4d8c (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250116 02:58:12.554145 20798 raft_consensus.cc:357] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.555213 20798 raft_consensus.cc:383] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:12.555578 20798 raft_consensus.cc:738] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:12.556463 20798 consensus_queue.cc:260] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.569620 20598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31db17a68ff141f3bd799ccc59c81fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.581557 20708 heartbeater.cc:501] Master 127.18.238.190:37551 was elected leader, sending a full tablet report...
I20250116 02:58:12.588328 20741 tablet_service.cc:1467] Processing CreateTablet for tablet bddd1f65156348fcb619edd31a8971f8 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250116 02:58:12.589589 20741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddd1f65156348fcb619edd31a8971f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.594714 20589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8115c09ba24b95a9969edcc6ca4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.598316 20799 ts_tablet_manager.cc:1428] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.233s user 0.024s sys 0.016s
I20250116 02:58:12.599627 20799 tablet_bootstrap.cc:492] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:12.599342 20669 tablet_service.cc:1467] Processing CreateTablet for tablet 99adad9dddaa4121a293aef6fccd1694 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250116 02:58:12.600930 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99adad9dddaa4121a293aef6fccd1694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.606348 20740 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8e1b8edadd42f08808ffaf11b40c8e (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250116 02:58:12.607789 20740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8e1b8edadd42f08808ffaf11b40c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.584260 20747 tablet_service.cc:1467] Processing CreateTablet for tablet 03c04d7691954d8e9278d9ba71f359fe (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250116 02:58:12.584600 20744 tablet_service.cc:1467] Processing CreateTablet for tablet 2960f90503b44912b4b4b58cd2ceb89f (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250116 02:58:12.609014 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7658a3a42c44f485085e75a4bd7b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.621160 20804 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:12.621757 20804 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.635416 20592 tablet_service.cc:1467] Processing CreateTablet for tablet d92adda8583d41508be4e5a8002cac4d (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250116 02:58:12.644989 20797 tablet_bootstrap.cc:492] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:12.645596 20797 ts_tablet_manager.cc:1397] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.271s user 0.013s sys 0.010s
I20250116 02:58:12.644915 20743 tablet_service.cc:1467] Processing CreateTablet for tablet 0b57de7b372d4101b7f88debce8eea57 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250116 02:58:12.647519 20668 tablet_service.cc:1467] Processing CreateTablet for tablet 03c04d7691954d8e9278d9ba71f359fe (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250116 02:58:12.652495 20668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c04d7691954d8e9278d9ba71f359fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.652927 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92adda8583d41508be4e5a8002cac4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.657179 20804 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:12.661854 20797 raft_consensus.cc:357] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.677585 20747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c04d7691954d8e9278d9ba71f359fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.664866 20797 raft_consensus.cc:383] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:12.681110 20797 raft_consensus.cc:738] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:12.691773 20798 ts_tablet_manager.cc:1428] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.270s user 0.016s sys 0.018s
I20250116 02:58:12.693158 20798 tablet_bootstrap.cc:492] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:12.698421 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2960f90503b44912b4b4b58cd2ceb89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.727456 20672 tablet_service.cc:1467] Processing CreateTablet for tablet e1b6569e42b74436b3ec62c56499eb17 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250116 02:58:12.728194 20742 tablet_service.cc:1467] Processing CreateTablet for tablet 79064d60f0d74dc4bb2733272417faf8 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250116 02:58:12.732703 20664 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8e1b8edadd42f08808ffaf11b40c8e (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250116 02:58:12.735746 20799 tablet_bootstrap.cc:654] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:12.735348 20593 tablet_service.cc:1467] Processing CreateTablet for tablet b0d2558ee3674a5889434a45c66ac3c1 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250116 02:58:12.736838 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d2558ee3674a5889434a45c66ac3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.738590 20742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79064d60f0d74dc4bb2733272417faf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.712733 20739 tablet_service.cc:1467] Processing CreateTablet for tablet e1b6569e42b74436b3ec62c56499eb17 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250116 02:58:12.740048 20739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1b6569e42b74436b3ec62c56499eb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.744547 20738 tablet_service.cc:1467] Processing CreateTablet for tablet 7c32cae59d8843b993750c4fe4e543e0 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250116 02:58:12.751003 20671 tablet_service.cc:1467] Processing CreateTablet for tablet 79064d60f0d74dc4bb2733272417faf8 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250116 02:58:12.757550 20670 tablet_service.cc:1467] Processing CreateTablet for tablet 7c32cae59d8843b993750c4fe4e543e0 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250116 02:58:12.758893 20590 tablet_service.cc:1467] Processing CreateTablet for tablet 0b57de7b372d4101b7f88debce8eea57 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250116 02:58:12.682096 20797 consensus_queue.cc:260] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.760577 20667 tablet_service.cc:1467] Processing CreateTablet for tablet e8dd85faff294418bdf7bf8a44f21ea0 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250116 02:58:12.768836 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b57de7b372d4101b7f88debce8eea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.769894 20664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8e1b8edadd42f08808ffaf11b40c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.705312 20591 tablet_service.cc:1467] Processing CreateTablet for tablet eb87db38c5904a70bb4b4b6d11afa1ca (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250116 02:58:12.784755 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:12.786542 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b57de7b372d4101b7f88debce8eea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.786162 20665 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba18c5d216643cbb5689397ab22da23 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250116 02:58:12.791285 20741 tablet_service.cc:1467] Processing CreateTablet for tablet e8dd85faff294418bdf7bf8a44f21ea0 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250116 02:58:12.792668 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79064d60f0d74dc4bb2733272417faf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.794548 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1b6569e42b74436b3ec62c56499eb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.795845 20665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba18c5d216643cbb5689397ab22da23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.796772 20669 tablet_service.cc:1467] Processing CreateTablet for tablet a569f56f9d2143869b29c78321047d00 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250116 02:58:12.798015 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a569f56f9d2143869b29c78321047d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.798904 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8dd85faff294418bdf7bf8a44f21ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.799054 20797 ts_tablet_manager.cc:1428] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.153s user 0.009s sys 0.000s
I20250116 02:58:12.800258 20797 tablet_bootstrap.cc:492] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:12.803258 20673 tablet_service.cc:1467] Processing CreateTablet for tablet 2960f90503b44912b4b4b58cd2ceb89f (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250116 02:58:12.804816 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2960f90503b44912b4b4b58cd2ceb89f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250116 02:58:12.806320 20709 tablet.cc:2367] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250116 02:58:12.761416 20666 tablet_service.cc:1467] Processing CreateTablet for tablet f929439d9fe44e75b91fde95f7953378 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250116 02:58:12.807705 20666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f929439d9fe44e75b91fde95f7953378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.809459 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c32cae59d8843b993750c4fe4e543e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.811484 20597 tablet_service.cc:1467] Processing CreateTablet for tablet 99adad9dddaa4121a293aef6fccd1694 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250116 02:58:12.813073 20597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99adad9dddaa4121a293aef6fccd1694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.825457 20798 tablet_bootstrap.cc:654] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:12.826640 20745 tablet_service.cc:1467] Processing CreateTablet for tablet f929439d9fe44e75b91fde95f7953378 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250116 02:58:12.832862 20591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb87db38c5904a70bb4b4b6d11afa1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.840957 20741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8dd85faff294418bdf7bf8a44f21ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.842542 20738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c32cae59d8843b993750c4fe4e543e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.850004 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f929439d9fe44e75b91fde95f7953378. 1 dirs total, 0 dirs full, 0 dirs failed
W20250116 02:58:12.852583 20571 leader_election.cc:343] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:12.858243 20810 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:12.858934 20810 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.942060 20797 tablet_bootstrap.cc:654] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:12.942533 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:12.944569 20723 leader_election.cc:343] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:12.952812 20798 tablet_bootstrap.cc:492] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:12.958158 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:12.960498 20572 leader_election.cc:343] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:12.961040 20572 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:12.961658 20596 tablet_service.cc:1467] Processing CreateTablet for tablet 03c04d7691954d8e9278d9ba71f359fe (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250116 02:58:12.963608 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c04d7691954d8e9278d9ba71f359fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.967185 20799 tablet_bootstrap.cc:492] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:12.967870 20799 ts_tablet_manager.cc:1397] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.368s user 0.012s sys 0.010s
I20250116 02:58:12.970989 20803 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:12.970995 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:58:12.974054 20720 leader_election.cc:343] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:12.974543 20720 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:12.975708 20804 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:12.953418 20798 ts_tablet_manager.cc:1397] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.260s user 0.012s sys 0.014s
I20250116 02:58:12.985875 20799 raft_consensus.cc:357] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:12.987115 20810 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:12.946390 20592 tablet_service.cc:1467] Processing CreateTablet for tablet bddd1f65156348fcb619edd31a8971f8 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250116 02:58:12.993672 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddd1f65156348fcb619edd31a8971f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:12.986992 20799 raft_consensus.cc:383] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.000877 20799 raft_consensus.cc:738] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.017832 20672 tablet_service.cc:1467] Processing CreateTablet for tablet b799fc79422648ca95fd3d30ab97fe28 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250116 02:58:13.018675 20742 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba18c5d216643cbb5689397ab22da23 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250116 02:58:13.018857 20668 tablet_service.cc:1467] Processing CreateTablet for tablet 65646a05d3974fe8bd8e187dc68a5a05 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250116 02:58:13.019450 20799 consensus_queue.cc:260] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.043709 20742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba18c5d216643cbb5689397ab22da23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.045267 20799 ts_tablet_manager.cc:1428] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.077s user 0.005s sys 0.004s
I20250116 02:58:13.021579 20594 tablet_service.cc:1467] Processing CreateTablet for tablet 2960f90503b44912b4b4b58cd2ceb89f (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250116 02:58:13.046751 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2960f90503b44912b4b4b58cd2ceb89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.050671 20593 tablet_service.cc:1467] Processing CreateTablet for tablet e1b6569e42b74436b3ec62c56499eb17 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250116 02:58:13.051960 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1b6569e42b74436b3ec62c56499eb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.054827 20799 tablet_bootstrap.cc:492] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:13.032656 20744 tablet_service.cc:1467] Processing CreateTablet for tablet a569f56f9d2143869b29c78321047d00 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250116 02:58:13.033972 20595 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8e1b8edadd42f08808ffaf11b40c8e (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250116 02:58:13.042479 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b799fc79422648ca95fd3d30ab97fe28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.068384 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a569f56f9d2143869b29c78321047d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.061355 20803 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:13.069535 20803 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.071647 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8e1b8edadd42f08808ffaf11b40c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.072997 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:58:13.074920 20571 leader_election.cc:343] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.078200 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:13.080329 20572 leader_election.cc:343] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.080875 20572 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:13.084239 20668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65646a05d3974fe8bd8e187dc68a5a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.054517 20739 tablet_service.cc:1467] Processing CreateTablet for tablet b799fc79422648ca95fd3d30ab97fe28 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250116 02:58:13.092269 20739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b799fc79422648ca95fd3d30ab97fe28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.082942 20803 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:13.095249 20803 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:13.066154 20747 tablet_service.cc:1467] Processing CreateTablet for tablet 54535b75280f444ca16b17d3f05cf9eb (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250116 02:58:13.099627 20798 raft_consensus.cc:357] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.100379 20747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54535b75280f444ca16b17d3f05cf9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.100479 20798 raft_consensus.cc:383] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.100876 20798 raft_consensus.cc:738] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.101656 20798 consensus_queue.cc:260] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.086870 20799 tablet_bootstrap.cc:654] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.111325 20746 tablet_service.cc:1467] Processing CreateTablet for tablet ff78e82fe8da4dd58822fb1d565eb3be (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250116 02:58:13.112810 20746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff78e82fe8da4dd58822fb1d565eb3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.120205 20745 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3fee7b273345558c661eda0710029c (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250116 02:58:13.121767 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d3fee7b273345558c661eda0710029c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.125303 20590 tablet_service.cc:1467] Processing CreateTablet for tablet 7c32cae59d8843b993750c4fe4e543e0 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250116 02:58:13.132071 20798 ts_tablet_manager.cc:1428] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.147s user 0.007s sys 0.000s
I20250116 02:58:13.131721 20593 tablet_service.cc:1467] Processing CreateTablet for tablet f929439d9fe44e75b91fde95f7953378 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250116 02:58:13.133198 20798 tablet_bootstrap.cc:492] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
W20250116 02:58:13.137213 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 56934.2us
I20250116 02:58:13.114125 20597 tablet_service.cc:1467] Processing CreateTablet for tablet 79064d60f0d74dc4bb2733272417faf8 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250116 02:58:13.155254 20797 tablet_bootstrap.cc:492] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:13.155864 20797 ts_tablet_manager.cc:1397] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.356s user 0.017s sys 0.021s
I20250116 02:58:13.157892 20671 tablet_service.cc:1467] Processing CreateTablet for tablet 54535b75280f444ca16b17d3f05cf9eb (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250116 02:58:13.159207 20797 raft_consensus.cc:357] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.164000 20797 raft_consensus.cc:383] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.164376 20797 raft_consensus.cc:738] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.165299 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54535b75280f444ca16b17d3f05cf9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.165694 20797 consensus_queue.cc:260] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.171917 20598 tablet_service.cc:1467] Processing CreateTablet for tablet a569f56f9d2143869b29c78321047d00 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250116 02:58:13.126241 20589 tablet_service.cc:1467] Processing CreateTablet for tablet e8dd85faff294418bdf7bf8a44f21ea0 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250116 02:58:13.179512 20740 tablet_service.cc:1467] Processing CreateTablet for tablet d9815466261546de9d937eeb8cd5cd2e (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250116 02:58:13.180989 20740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9815466261546de9d937eeb8cd5cd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.174196 20665 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3fee7b273345558c661eda0710029c (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250116 02:58:13.183081 20665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d3fee7b273345558c661eda0710029c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.181175 20594 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba18c5d216643cbb5689397ab22da23 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250116 02:58:13.185824 20797 ts_tablet_manager.cc:1428] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.030s user 0.006s sys 0.003s
I20250116 02:58:13.187662 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:13.192695 20649 leader_election.cc:343] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.189116 20743 tablet_service.cc:1467] Processing CreateTablet for tablet 65646a05d3974fe8bd8e187dc68a5a05 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250116 02:58:13.199909 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65646a05d3974fe8bd8e187dc68a5a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.149281 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c32cae59d8843b993750c4fe4e543e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.194201 20741 tablet_service.cc:1467] Processing CreateTablet for tablet 81fd75864c6947f09d4e5d31a757c4fa (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250116 02:58:13.207379 20741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81fd75864c6947f09d4e5d31a757c4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.208371 20664 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd4be964f5040df93f3e609dddcc60a (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250116 02:58:13.215477 20598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a569f56f9d2143869b29c78321047d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.216313 20597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79064d60f0d74dc4bb2733272417faf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.218230 20589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8dd85faff294418bdf7bf8a44f21ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.220762 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba18c5d216643cbb5689397ab22da23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.223470 20592 tablet_service.cc:1467] Processing CreateTablet for tablet 65646a05d3974fe8bd8e187dc68a5a05 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250116 02:58:13.224956 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65646a05d3974fe8bd8e187dc68a5a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.226509 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f929439d9fe44e75b91fde95f7953378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.227517 20667 tablet_service.cc:1467] Processing CreateTablet for tablet ff78e82fe8da4dd58822fb1d565eb3be (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250116 02:58:13.231475 20591 tablet_service.cc:1467] Processing CreateTablet for tablet b799fc79422648ca95fd3d30ab97fe28 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250116 02:58:13.233107 20591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b799fc79422648ca95fd3d30ab97fe28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.235509 20664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd4be964f5040df93f3e609dddcc60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.238938 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff78e82fe8da4dd58822fb1d565eb3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.240226 20669 tablet_service.cc:1467] Processing CreateTablet for tablet 81fd75864c6947f09d4e5d31a757c4fa (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250116 02:58:13.241672 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81fd75864c6947f09d4e5d31a757c4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.244165 20668 tablet_service.cc:1467] Processing CreateTablet for tablet 19c9bd4e72374377958093c605e5c6ae (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250116 02:58:13.245545 20668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c9bd4e72374377958093c605e5c6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.251583 20742 tablet_service.cc:1467] Processing CreateTablet for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250116 02:58:13.253320 20742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.253592 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:13.254387 20757 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:13.255879 20797 tablet_bootstrap.cc:492] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:13.255733 20673 tablet_service.cc:1467] Processing CreateTablet for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250116 02:58:13.256021 20647 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:13.258845 20810 raft_consensus.cc:2798] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:13.259254 20810 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:13.259614 20810 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:13.230010 20670 tablet_service.cc:1467] Processing CreateTablet for tablet d9815466261546de9d937eeb8cd5cd2e (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250116 02:58:13.263535 20798 tablet_bootstrap.cc:654] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.269127 20810 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.279771 20799 tablet_bootstrap.cc:492] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:13.280411 20799 ts_tablet_manager.cc:1397] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.226s user 0.004s sys 0.009s
I20250116 02:58:13.283605 20799 raft_consensus.cc:357] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.284441 20799 raft_consensus.cc:383] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.284765 20799 raft_consensus.cc:738] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.285559 20799 consensus_queue.cc:260] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:13.289994 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:13.302932 20747 tablet_service.cc:1467] Processing CreateTablet for tablet 38583cae882144f48cec12c72ba697cf (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250116 02:58:13.314561 20747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38583cae882144f48cec12c72ba697cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.314425 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:58:13.316390 20649 leader_election.cc:343] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.317409 20820 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:13.318008 20820 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.319206 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:13.319872 20757 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:13.322209 20820 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:13.325585 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:13.332228 20799 ts_tablet_manager.cc:1428] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.051s user 0.006s sys 0.002s
I20250116 02:58:13.333019 20744 tablet_service.cc:1467] Processing CreateTablet for tablet fe9318d6bdac457facfcaa97dea377f3 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20250116 02:58:13.333969 20571 leader_election.cc:343] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.336028 20672 tablet_service.cc:1467] Processing CreateTablet for tablet 38583cae882144f48cec12c72ba697cf (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250116 02:58:13.338795 20666 tablet_service.cc:1467] Processing CreateTablet for tablet 221670297d85466bac4e5a9bfead9b18 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250116 02:58:13.320542 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.347313 20799 tablet_bootstrap.cc:492] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:13.330487 20746 tablet_service.cc:1467] Processing CreateTablet for tablet b626139c64c14e309e332bc7605ba6f1 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250116 02:58:13.332150 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:13.356488 20572 leader_election.cc:343] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.357019 20572 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:13.358392 20820 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:13.363199 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38583cae882144f48cec12c72ba697cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.365710 20666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221670297d85466bac4e5a9bfead9b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.366286 20596 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43834 (request call id 30) took 1062 ms. Trace:
I20250116 02:58:13.323410 20810 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:13.371093 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9815466261546de9d937eeb8cd5cd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.302274 20738 tablet_service.cc:1467] Processing CreateTablet for tablet 221670297d85466bac4e5a9bfead9b18 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250116 02:58:13.383198 20594 tablet_service.cc:1467] Processing CreateTablet for tablet 54535b75280f444ca16b17d3f05cf9eb (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250116 02:58:13.385607 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9318d6bdac457facfcaa97dea377f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.386585 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54535b75280f444ca16b17d3f05cf9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.376186 20757 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:13.392604 20595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43834 (request call id 32) took 1059 ms. Trace:
I20250116 02:58:13.393819 20739 tablet_service.cc:1467] Processing CreateTablet for tablet 59601d2a275b4efdbbc89dbcb608871c (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250116 02:58:13.401436 20647 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:13.393819 20747 tablet_service.cc:1467] Processing CreateTablet for tablet 19c9bd4e72374377958093c605e5c6ae (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250116 02:58:13.402769 20810 raft_consensus.cc:2798] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:13.403131 20671 tablet_service.cc:1467] Processing CreateTablet for tablet b626139c64c14e309e332bc7605ba6f1 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250116 02:58:13.401684 20595 rpcz_store.cc:276] 0116 02:58:12.332865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:58:13.033066 (+700201us) service_pool.cc:225] Handling call
0116 02:58:13.071160 (+ 38094us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0116 02:58:13.071269 (+ 109us) ts_tablet_manager.cc:676] Creating new metadata...
0116 02:58:13.239769 (+168500us) tablet_metadata.cc:641] Metadata flushed
0116 02:58:13.392024 (+152255us) tablet_replica.cc:206] Creating consensus instance
0116 02:58:13.392531 (+ 507us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7022}
I20250116 02:58:13.408013 20669 tablet_service.cc:1467] Processing CreateTablet for tablet fe9318d6bdac457facfcaa97dea377f3 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250116 02:58:13.397017 20590 tablet_service.cc:1467] Processing CreateTablet for tablet ff78e82fe8da4dd58822fb1d565eb3be (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250116 02:58:13.397670 20743 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef2f3fbb2a04b7fb68c0bd275697181 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250116 02:58:13.398366 20746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b626139c64c14e309e332bc7605ba6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.396123 20741 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd4be964f5040df93f3e609dddcc60a (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250116 02:58:13.413594 20664 tablet_service.cc:1467] Processing CreateTablet for tablet 59601d2a275b4efdbbc89dbcb608871c (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250116 02:58:13.402797 20598 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3fee7b273345558c661eda0710029c (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250116 02:58:13.416079 20741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd4be964f5040df93f3e609dddcc60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.429677 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef2f3fbb2a04b7fb68c0bd275697181. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.432546 20798 tablet_bootstrap.cc:492] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:13.404616 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b626139c64c14e309e332bc7605ba6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.401721 20591 tablet_service.cc:1467] Processing CreateTablet for tablet d9815466261546de9d937eeb8cd5cd2e (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250116 02:58:13.437037 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff78e82fe8da4dd58822fb1d565eb3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.366900 20596 rpcz_store.cc:276] 0116 02:58:12.303595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:58:12.950224 (+646629us) service_pool.cc:225] Handling call
0116 02:58:12.962827 (+ 12603us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0116 02:58:12.962976 (+ 149us) ts_tablet_manager.cc:676] Creating new metadata...
0116 02:58:13.162391 (+199415us) tablet_metadata.cc:641] Metadata flushed
0116 02:58:13.365619 (+203228us) tablet_replica.cc:206] Creating consensus instance
0116 02:58:13.366208 (+ 589us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:58:13.396180 20742 tablet_service.cc:1467] Processing CreateTablet for tablet f7c662e0a6684e369f470f73b4bfcd80 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250116 02:58:13.409508 20595 tablet_service.cc:1467] Processing CreateTablet for tablet 81fd75864c6947f09d4e5d31a757c4fa (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250116 02:58:13.390758 20797 tablet_bootstrap.cc:700] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f: Time spent opening tablet: real 0.101s user 0.001s sys 0.001s
I20250116 02:58:13.433173 20798 ts_tablet_manager.cc:1397] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.300s user 0.009s sys 0.024s
I20250116 02:58:13.435057 20739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59601d2a275b4efdbbc89dbcb608871c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.445367 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9318d6bdac457facfcaa97dea377f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.457830 20798 raft_consensus.cc:357] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.458953 20798 raft_consensus.cc:383] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.459297 20798 raft_consensus.cc:738] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.449537 20596 tablet_service.cc:1467] Processing CreateTablet for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250116 02:58:13.465598 20598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d3fee7b273345558c661eda0710029c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.466169 20799 tablet_bootstrap.cc:654] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.471733 20747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c9bd4e72374377958093c605e5c6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.470443 20797 tablet_bootstrap.cc:654] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.473060 20597 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43834 (request call id 34) took 1090 ms. Trace:
I20250116 02:58:13.468772 20810 raft_consensus.cc:695] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:13.471529 20804 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:13.477311 20597 rpcz_store.cc:276] 0116 02:58:12.382564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:58:13.113191 (+730627us) service_pool.cc:225] Handling call
0116 02:58:13.147957 (+ 34766us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0116 02:58:13.148106 (+ 149us) ts_tablet_manager.cc:676] Creating new metadata...
0116 02:58:13.296863 (+148757us) tablet_metadata.cc:641] Metadata flushed
0116 02:58:13.433290 (+136427us) tablet_replica.cc:206] Creating consensus instance
0116 02:58:13.471593 (+ 38303us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38902}
I20250116 02:58:13.489455 20804 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.490846 20597 tablet_service.cc:1467] Processing CreateTablet for tablet 221670297d85466bac4e5a9bfead9b18 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250116 02:58:13.491859 20804 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:13.469887 20589 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43834 (request call id 36) took 1010 ms. Trace:
I20250116 02:58:13.492233 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81fd75864c6947f09d4e5d31a757c4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.492377 20589 rpcz_store.cc:276] 0116 02:58:12.459292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:58:13.125314 (+666022us) service_pool.cc:225] Handling call
0116 02:58:13.176625 (+ 51311us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0116 02:58:13.176735 (+ 110us) ts_tablet_manager.cc:676] Creating new metadata...
0116 02:58:13.344682 (+167947us) tablet_metadata.cc:641] Metadata flushed
0116 02:58:13.427629 (+ 82947us) tablet_replica.cc:206] Creating consensus instance
0116 02:58:13.468000 (+ 40371us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:58:13.493885 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.491859 20592 tablet_service.cc:1467] Processing CreateTablet for tablet b626139c64c14e309e332bc7605ba6f1 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250116 02:58:13.494637 20589 tablet_service.cc:1467] Processing CreateTablet for tablet fe9318d6bdac457facfcaa97dea377f3 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250116 02:58:13.480580 20823 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:13.460250 20798 consensus_queue.cc:260] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.477667 20664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59601d2a275b4efdbbc89dbcb608871c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.469653 20745 tablet_service.cc:1467] Processing CreateTablet for tablet 96dc04669716445ab04931358be7efbf (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250116 02:58:13.477372 20667 tablet_service.cc:1467] Processing CreateTablet for tablet f7c662e0a6684e369f470f73b4bfcd80 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250116 02:58:13.504997 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c662e0a6684e369f470f73b4bfcd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.487946 20593 tablet_service.cc:1467] Processing CreateTablet for tablet 38583cae882144f48cec12c72ba697cf (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250116 02:58:13.493803 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:13.507028 20723 leader_election.cc:343] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.493214 20665 tablet_service.cc:1467] Processing CreateTablet for tablet 96dc04669716445ab04931358be7efbf (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250116 02:58:13.467156 20666 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef2f3fbb2a04b7fb68c0bd275697181 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250116 02:58:13.498528 20591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9815466261546de9d937eeb8cd5cd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.512190 20597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221670297d85466bac4e5a9bfead9b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.515486 20666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef2f3fbb2a04b7fb68c0bd275697181. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.497318 20823 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.480005 20740 tablet_service.cc:1467] Processing CreateTablet for tablet 8229db04eecb4bc3a9ff24fc8bcac934 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250116 02:58:13.519593 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:13.520125 20823 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:13.521116 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:13.521883 20757 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:13.523419 20647 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:13.524813 20810 consensus_queue.cc:237] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.529690 20670 tablet_service.cc:1467] Processing CreateTablet for tablet 8229db04eecb4bc3a9ff24fc8bcac934 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250116 02:58:13.531370 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:13.533617 20649 leader_election.cc:343] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.535073 20589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9318d6bdac457facfcaa97dea377f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.535172 20823 raft_consensus.cc:2798] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:13.535624 20823 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:13.535989 20823 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:13.538872 20738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221670297d85466bac4e5a9bfead9b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.539577 20740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8229db04eecb4bc3a9ff24fc8bcac934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.540503 20665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96dc04669716445ab04931358be7efbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.539832 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38583cae882144f48cec12c72ba697cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.542593 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96dc04669716445ab04931358be7efbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.543833 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8229db04eecb4bc3a9ff24fc8bcac934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.551144 20742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c662e0a6684e369f470f73b4bfcd80. 1 dirs total, 0 dirs full, 0 dirs failed
W20250116 02:58:13.554781 20720 leader_election.cc:343] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.557667 20720 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:13.558995 20804 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:13.557201 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b626139c64c14e309e332bc7605ba6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.567224 20798 ts_tablet_manager.cc:1428] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.113s user 0.008s sys 0.000s
I20250116 02:58:13.568560 20798 tablet_bootstrap.cc:492] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:13.595971 20823 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.604065 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:58:13.605902 20649 leader_election.cc:343] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.607123 20823 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:13.608484 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:13.609158 20757 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:13.607847 20594 tablet_service.cc:1467] Processing CreateTablet for tablet 59601d2a275b4efdbbc89dbcb608871c (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250116 02:58:13.632813 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59601d2a275b4efdbbc89dbcb608871c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.644351 20757 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:13.646871 20590 tablet_service.cc:1467] Processing CreateTablet for tablet 19c9bd4e72374377958093c605e5c6ae (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250116 02:58:13.648356 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c9bd4e72374377958093c605e5c6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.648892 20505 catalog_manager.cc:5526] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:13.656380 20595 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd4be964f5040df93f3e609dddcc60a (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250116 02:58:13.657368 20797 tablet_bootstrap.cc:492] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:13.658507 20797 ts_tablet_manager.cc:1397] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.403s user 0.010s sys 0.008s
I20250116 02:58:13.657959 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd4be964f5040df93f3e609dddcc60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.668279 20647 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:13.677803 20798 tablet_bootstrap.cc:654] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.671675 20810 raft_consensus.cc:2798] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:13.684779 20810 raft_consensus.cc:695] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:13.678332 20797 raft_consensus.cc:357] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:58:13.688854 20797 raft_consensus.cc:383] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.688894 20591 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef2f3fbb2a04b7fb68c0bd275697181 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250116 02:58:13.690258 20591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef2f3fbb2a04b7fb68c0bd275697181. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.685823 20810 consensus_queue.cc:237] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.699478 20799 tablet_bootstrap.cc:492] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:13.700115 20799 ts_tablet_manager.cc:1397] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.353s user 0.012s sys 0.020s
I20250116 02:58:13.689291 20797 raft_consensus.cc:738] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.703562 20597 tablet_service.cc:1467] Processing CreateTablet for tablet 96dc04669716445ab04931358be7efbf (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250116 02:58:13.707896 20597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96dc04669716445ab04931358be7efbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.718196 20799 raft_consensus.cc:357] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.719161 20799 raft_consensus.cc:383] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.718837 20797 consensus_queue.cc:260] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:58:13.719463 20799 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.721283 20799 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.722014 20797 ts_tablet_manager.cc:1428] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20250116 02:58:13.730363 20799 ts_tablet_manager.cc:1428] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.030s user 0.004s sys 0.004s
I20250116 02:58:13.736629 20799 tablet_bootstrap.cc:492] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:13.743853 20797 tablet_bootstrap.cc:492] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:13.753167 20592 tablet_service.cc:1467] Processing CreateTablet for tablet 8229db04eecb4bc3a9ff24fc8bcac934 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250116 02:58:13.686448 20596 tablet_service.cc:1467] Processing CreateTablet for tablet f7c662e0a6684e369f470f73b4bfcd80 (DEFAULT_TABLE table=test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250116 02:58:13.768414 20798 tablet_bootstrap.cc:492] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:13.769152 20798 ts_tablet_manager.cc:1397] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.201s user 0.017s sys 0.009s
I20250116 02:58:13.776140 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8229db04eecb4bc3a9ff24fc8bcac934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.789315 20797 tablet_bootstrap.cc:654] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.790158 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c662e0a6684e369f470f73b4bfcd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:13.809000 20798 raft_consensus.cc:357] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.809942 20798 raft_consensus.cc:383] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.810268 20798 raft_consensus.cc:738] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.812569 20799 tablet_bootstrap.cc:654] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.812552 20798 consensus_queue.cc:260] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.815594 20798 ts_tablet_manager.cc:1428] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.046s user 0.004s sys 0.004s
I20250116 02:58:13.816794 20798 tablet_bootstrap.cc:492] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:13.816704 20797 tablet_bootstrap.cc:492] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:13.819700 20797 ts_tablet_manager.cc:1397] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.004s
I20250116 02:58:13.824187 20797 raft_consensus.cc:357] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.825138 20797 raft_consensus.cc:383] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.825464 20797 raft_consensus.cc:738] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.826375 20797 consensus_queue.cc:260] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.829551 20797 ts_tablet_manager.cc:1428] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250116 02:58:13.830946 20797 tablet_bootstrap.cc:492] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:13.831543 20798 tablet_bootstrap.cc:654] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.837159 20797 tablet_bootstrap.cc:654] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.837360 20506 catalog_manager.cc:5526] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:13.840472 20799 tablet_bootstrap.cc:492] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:13.841092 20799 ts_tablet_manager.cc:1397] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.009s
I20250116 02:58:13.844864 20799 raft_consensus.cc:357] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.846505 20799 raft_consensus.cc:383] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.846987 20799 raft_consensus.cc:738] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.847115 20798 tablet_bootstrap.cc:492] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:13.847514 20820 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:13.847824 20798 ts_tablet_manager.cc:1397] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.002s
I20250116 02:58:13.848053 20820 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.848153 20799 consensus_queue.cc:260] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.851406 20799 ts_tablet_manager.cc:1428] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20250116 02:58:13.851989 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:58:13.853770 20571 leader_election.cc:343] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.854504 20820 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:13.855530 20798 raft_consensus.cc:357] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.856432 20798 raft_consensus.cc:383] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.856740 20798 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.856946 20797 tablet_bootstrap.cc:492] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:13.857501 20797 ts_tablet_manager.cc:1397] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.005s
I20250116 02:58:13.857291 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:13.857699 20798 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:13.858971 20572 leader_election.cc:343] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.859189 20799 tablet_bootstrap.cc:492] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:13.860412 20798 ts_tablet_manager.cc:1428] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250116 02:58:13.862656 20572 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:13.863610 20798 tablet_bootstrap.cc:492] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:13.864663 20820 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:13.866708 20799 tablet_bootstrap.cc:654] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.868502 20810 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:13.868933 20810 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.862344 20797 raft_consensus.cc:357] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.870723 20798 tablet_bootstrap.cc:654] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.870805 20810 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:13.872047 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:13.872859 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:13.873441 20608 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:13.871999 20797 raft_consensus.cc:383] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250116 02:58:13.874915 20647 leader_election.cc:343] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.875128 20797 raft_consensus.cc:738] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.876206 20797 consensus_queue.cc:260] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.877338 20649 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:13.879442 20797 ts_tablet_manager.cc:1428] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.022s user 0.005s sys 0.002s
I20250116 02:58:13.880175 20810 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:13.880653 20810 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:13.881016 20810 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:13.884343 20797 tablet_bootstrap.cc:492] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:13.886226 20798 tablet_bootstrap.cc:492] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:13.886921 20798 ts_tablet_manager.cc:1397] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20250116 02:58:13.889539 20810 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.890293 20799 tablet_bootstrap.cc:492] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:13.890328 20798 raft_consensus.cc:357] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.892046 20810 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:13.891738 20798 raft_consensus.cc:383] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.893231 20798 raft_consensus.cc:738] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.893401 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:13.894240 20797 tablet_bootstrap.cc:654] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.894574 20798 consensus_queue.cc:260] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:13.895133 20647 leader_election.cc:343] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.895795 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:13.896442 20608 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:13.890894 20799 ts_tablet_manager.cc:1397] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.001s
I20250116 02:58:13.899964 20799 raft_consensus.cc:357] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.900584 20799 raft_consensus.cc:383] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.900887 20799 raft_consensus.cc:738] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.901772 20799 consensus_queue.cc:260] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.903316 20797 tablet_bootstrap.cc:492] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:13.903920 20797 ts_tablet_manager.cc:1397] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.005s
I20250116 02:58:13.906009 20798 ts_tablet_manager.cc:1428] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.019s user 0.001s sys 0.006s
I20250116 02:58:13.905856 20608 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:13.907294 20649 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:13.909807 20798 tablet_bootstrap.cc:492] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:13.913604 20799 ts_tablet_manager.cc:1428] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250116 02:58:13.908195 20810 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:13.915517 20810 raft_consensus.cc:695] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:13.907577 20797 raft_consensus.cc:357] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.916806 20797 raft_consensus.cc:383] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.916477 20810 consensus_queue.cc:237] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.917161 20797 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.918131 20797 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.921183 20797 ts_tablet_manager.cc:1428] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250116 02:58:13.915674 20799 tablet_bootstrap.cc:492] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:13.921927 20798 tablet_bootstrap.cc:654] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.923674 20797 tablet_bootstrap.cc:492] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:13.928759 20799 tablet_bootstrap.cc:654] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.934903 20797 tablet_bootstrap.cc:654] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.941663 20799 tablet_bootstrap.cc:492] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:13.942349 20799 ts_tablet_manager.cc:1397] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.002s
I20250116 02:58:13.945612 20799 raft_consensus.cc:357] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.946725 20799 raft_consensus.cc:383] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.947072 20799 raft_consensus.cc:738] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.948002 20799 consensus_queue.cc:260] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.949867 20797 tablet_bootstrap.cc:492] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:13.950364 20797 ts_tablet_manager.cc:1397] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.005s
I20250116 02:58:13.950767 20798 tablet_bootstrap.cc:492] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:13.951332 20798 ts_tablet_manager.cc:1397] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.042s user 0.028s sys 0.011s
I20250116 02:58:13.953207 20797 raft_consensus.cc:357] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.954027 20797 raft_consensus.cc:383] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.954375 20797 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.954100 20506 catalog_manager.cc:5526] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:13.954741 20798 raft_consensus.cc:357] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:58:13.955605 20798 raft_consensus.cc:383] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.955219 20797 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.955955 20798 raft_consensus.cc:738] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.956795 20799 ts_tablet_manager.cc:1428] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20250116 02:58:13.957455 20797 ts_tablet_manager.cc:1428] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250116 02:58:13.956827 20798 consensus_queue.cc:260] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:58:13.958123 20799 tablet_bootstrap.cc:492] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:13.959640 20798 ts_tablet_manager.cc:1428] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250116 02:58:13.960510 20797 tablet_bootstrap.cc:492] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:13.960583 20798 tablet_bootstrap.cc:492] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:13.965353 20799 tablet_bootstrap.cc:654] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.966681 20804 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:13.967437 20797 tablet_bootstrap.cc:654] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.967442 20798 tablet_bootstrap.cc:654] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:13.967206 20804 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.969403 20804 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:13.971294 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:13.972316 20608 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:13.973553 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:13.976537 20723 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:13.977331 20810 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:13.978086 20804 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:13.977891 20810 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.980190 20810 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:13.981457 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:13.981699 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:13.982436 20757 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
W20250116 02:58:13.983223 20649 leader_election.cc:343] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:13.984181 20647 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:13.985405 20810 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:13.985791 20810 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:13.986052 20797 tablet_bootstrap.cc:492] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:13.986683 20797 ts_tablet_manager.cc:1397] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.007s
I20250116 02:58:13.986158 20810 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:13.989591 20797 raft_consensus.cc:357] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.990356 20797 raft_consensus.cc:383] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.990670 20798 tablet_bootstrap.cc:492] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:13.990664 20797 raft_consensus.cc:738] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.991395 20798 ts_tablet_manager.cc:1397] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.005s
I20250116 02:58:13.992089 20804 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:13.991689 20797 consensus_queue.cc:260] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.994117 20823 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:13.994731 20823 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:58:13.995237 20798 raft_consensus.cc:357] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.996189 20798 raft_consensus.cc:383] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:13.996502 20798 raft_consensus.cc:738] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:13.996913 20823 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:13.997644 20797 ts_tablet_manager.cc:1428] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250116 02:58:13.997205 20798 consensus_queue.cc:260] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:13.998728 20797 tablet_bootstrap.cc:492] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:14.000098 20798 ts_tablet_manager.cc:1428] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250116 02:58:14.001186 20798 tablet_bootstrap.cc:492] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:14.002431 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.003258 20608 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:14.005482 20797 tablet_bootstrap.cc:654] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.005796 20649 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:14.006779 20823 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:14.007215 20823 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:14.007625 20823 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.008410 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:14.010216 20647 leader_election.cc:343] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.014377 20810 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.017781 20810 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.017972 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:14.019582 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:58:14.019824 20649 leader_election.cc:343] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.020298 20757 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.021852 20798 tablet_bootstrap.cc:654] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.022693 20799 tablet_bootstrap.cc:492] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.023240 20799 ts_tablet_manager.cc:1397] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.065s user 0.015s sys 0.001s
I20250116 02:58:14.026268 20799 raft_consensus.cc:357] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:58:14.026327 20823 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:58:14.027328 20799 raft_consensus.cc:383] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.027712 20799 raft_consensus.cc:738] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.028445 20799 consensus_queue.cc:260] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:58:14.030969 20799 ts_tablet_manager.cc:1428] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250116 02:58:14.031625 20757 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:14.033113 20647 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:14.034678 20824 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.037173 20824 raft_consensus.cc:695] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:14.037381 20608 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.038378 20824 consensus_queue.cc:237] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.045228 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:14.046017 20757 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.047473 20799 tablet_bootstrap.cc:492] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:14.048048 20850 consensus_queue.cc:1035] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.048605 20823 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:14.051855 20798 tablet_bootstrap.cc:492] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:14.051924 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:14.052573 20798 ts_tablet_manager.cc:1397] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.001s
I20250116 02:58:14.052690 20608 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.055768 20798 raft_consensus.cc:357] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.056622 20798 raft_consensus.cc:383] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.056923 20798 raft_consensus.cc:738] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.057301 20757 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:14.057771 20798 consensus_queue.cc:260] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.058902 20647 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:14.060074 20850 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.060837 20850 raft_consensus.cc:695] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:14.061880 20850 consensus_queue.cc:237] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:58:14.065662 20799 tablet_bootstrap.cc:654] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.068198 20798 ts_tablet_manager.cc:1428] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250116 02:58:14.068786 20608 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:14.079916 20505 catalog_manager.cc:5526] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:14.083631 20799 tablet_bootstrap.cc:492] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.084206 20799 ts_tablet_manager.cc:1397] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.004s
I20250116 02:58:14.088179 20799 raft_consensus.cc:357] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.089165 20799 raft_consensus.cc:383] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.089469 20799 raft_consensus.cc:738] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.069422 20798 tablet_bootstrap.cc:492] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:14.074414 20797 tablet_bootstrap.cc:492] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:14.090328 20797 ts_tablet_manager.cc:1397] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.092s user 0.019s sys 0.034s
I20250116 02:58:14.090283 20799 consensus_queue.cc:260] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.093525 20797 raft_consensus.cc:357] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.094370 20797 raft_consensus.cc:383] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.094719 20797 raft_consensus.cc:738] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.096025 20797 consensus_queue.cc:260] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.098591 20797 ts_tablet_manager.cc:1428] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20250116 02:58:14.099809 20797 tablet_bootstrap.cc:492] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:14.103091 20799 ts_tablet_manager.cc:1428] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.019s user 0.005s sys 0.003s
I20250116 02:58:14.108492 20850 consensus_queue.cc:1035] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.109010 20798 tablet_bootstrap.cc:654] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.112190 20799 tablet_bootstrap.cc:492] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:14.109030 20506 catalog_manager.cc:5526] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:14.120016 20799 tablet_bootstrap.cc:654] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.131544 20797 tablet_bootstrap.cc:654] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.136979 20798 tablet_bootstrap.cc:492] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:14.137486 20798 ts_tablet_manager.cc:1397] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.003s
I20250116 02:58:14.139110 20820 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.139580 20820 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.140162 20798 raft_consensus.cc:357] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.140825 20798 raft_consensus.cc:383] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.141077 20798 raft_consensus.cc:738] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.141502 20820 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.142230 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.143527 20797 tablet_bootstrap.cc:492] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:14.143579 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:14.144093 20571 leader_election.cc:343] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.144304 20797 ts_tablet_manager.cc:1397] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.004s
I20250116 02:58:14.144531 20757 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:14.145598 20572 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:14.146348 20820 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:14.146747 20820 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:14.147085 20820 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.147835 20797 raft_consensus.cc:357] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.148716 20797 raft_consensus.cc:383] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.149010 20797 raft_consensus.cc:738] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.149762 20797 consensus_queue.cc:260] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.150964 20798 consensus_queue.cc:260] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.153800 20798 ts_tablet_manager.cc:1428] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250116 02:58:14.156737 20820 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.159755 20820 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.160394 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:14.161077 20757 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.161616 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:58:14.163327 20571 leader_election.cc:343] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.165117 20798 tablet_bootstrap.cc:492] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:14.167837 20797 ts_tablet_manager.cc:1428] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.023s user 0.012s sys 0.000s
I20250116 02:58:14.168144 20757 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:14.169746 20799 tablet_bootstrap.cc:492] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.169538 20572 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:14.170269 20799 ts_tablet_manager.cc:1397] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.020s
I20250116 02:58:14.171278 20820 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.172394 20862 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.173425 20799 raft_consensus.cc:357] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.175400 20799 raft_consensus.cc:383] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.175755 20799 raft_consensus.cc:738] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.176447 20797 tablet_bootstrap.cc:492] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:14.176595 20799 consensus_queue.cc:260] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.179373 20799 ts_tablet_manager.cc:1428] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20250116 02:58:14.175863 20862 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.180609 20799 tablet_bootstrap.cc:492] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:14.181490 20862 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.182423 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.183194 20683 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:14.184645 20571 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:14.185688 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:14.185726 20862 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:14.187279 20799 tablet_bootstrap.cc:654] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.187834 20862 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:14.188241 20862 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:14.187333 20572 leader_election.cc:343] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.187283 20797 tablet_bootstrap.cc:654] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.198333 20804 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.198961 20804 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.200366 20798 tablet_bootstrap.cc:654] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.201185 20804 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:14.204720 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.204911 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.205821 20607 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
W20250116 02:58:14.206727 20720 leader_election.cc:343] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.207000 20862 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.207450 20723 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:14.208992 20804 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:14.209455 20804 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:14.209829 20804 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.210466 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:14.210466 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:14.212689 20683 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:14.213083 20572 leader_election.cc:343] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.211171 20820 raft_consensus.cc:695] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:14.217381 20820 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.216535 20799 tablet_bootstrap.cc:492] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.220542 20799 ts_tablet_manager.cc:1397] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.005s
I20250116 02:58:14.224048 20804 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.224555 20799 raft_consensus.cc:357] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.225750 20799 raft_consensus.cc:383] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.226176 20799 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.211357 20862 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.229189 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:14.229336 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:14.230296 20607 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.229739 20799 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:14.231276 20720 leader_election.cc:343] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.233270 20797 tablet_bootstrap.cc:492] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:14.233845 20797 ts_tablet_manager.cc:1397] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.023s
I20250116 02:58:14.236927 20797 raft_consensus.cc:357] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.238574 20797 raft_consensus.cc:383] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.239230 20798 tablet_bootstrap.cc:492] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:14.227588 20804 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:14.239408 20823 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.236820 20683 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:14.240082 20607 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:14.239953 20823 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.241390 20723 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:14.242224 20823 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.241421 20571 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:14.243882 20862 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.244416 20862 raft_consensus.cc:695] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:14.245355 20862 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.245275 20804 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.247905 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.252990 20608 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:14.249848 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:14.254817 20757 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:14.254716 20649 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:14.251927 20804 raft_consensus.cc:695] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:14.256172 20823 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:14.245920 20798 ts_tablet_manager.cc:1397] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.014s
I20250116 02:58:14.256669 20823 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:14.257125 20823 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.259094 20804 consensus_queue.cc:237] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.265818 20798 raft_consensus.cc:357] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.266947 20798 raft_consensus.cc:383] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.267268 20798 raft_consensus.cc:738] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.268087 20798 consensus_queue.cc:260] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.238945 20797 raft_consensus.cc:738] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.269796 20797 consensus_queue.cc:260] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.280179 20797 ts_tablet_manager.cc:1428] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.046s user 0.006s sys 0.003s
I20250116 02:58:14.282055 20506 catalog_manager.cc:5526] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:14.289999 20869 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.300002 20798 ts_tablet_manager.cc:1428] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.043s user 0.007s sys 0.001s
I20250116 02:58:14.301637 20798 tablet_bootstrap.cc:492] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:14.303501 20869 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.294523 20804 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.305745 20804 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.306728 20799 ts_tablet_manager.cc:1428] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.086s user 0.042s sys 0.017s
I20250116 02:58:14.306217 20504 catalog_manager.cc:5526] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:14.307835 20799 tablet_bootstrap.cc:492] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Bootstrap starting.
W20250116 02:58:14.294730 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:14.309742 20804 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:14.297367 20797 tablet_bootstrap.cc:492] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:14.311514 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.312402 20608 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:14.313580 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.314321 20683 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:14.314450 20799 tablet_bootstrap.cc:654] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.315220 20850 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.317842 20723 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:14.315729 20850 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.320487 20850 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.293872 20823 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.322629 20820 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.322976 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.323330 20820 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.324363 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:14.325567 20820 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:58:14.326227 20649 leader_election.cc:343] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.327595 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:58:14.329241 20571 leader_election.cc:343] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.330832 20608 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.335078 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:14.335947 20606 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:14.335892 20647 leader_election.cc:343] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.337034 20647 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:14.338444 20824 raft_consensus.cc:2743] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:14.339082 20824 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.339704 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.332429 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.340600 20683 raft_consensus.cc:2388] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:14.339519 20824 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.342757 20804 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:14.343268 20804 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:14.343678 20804 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.344781 20824 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.344650 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.335094 20823 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.331529 20869 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:14.366375 20607 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:14.368381 20649 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:14.369585 20824 raft_consensus.cc:2798] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:14.370055 20824 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:14.369920 20720 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:14.370564 20824 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.345448 20606 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:14.368250 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:14.371896 20850 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.372241 20649 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:14.372696 20757 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:14.373399 20810 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.374001 20810 raft_consensus.cc:695] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:14.375090 20810 consensus_queue.cc:237] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.381793 20869 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.386088 20798 tablet_bootstrap.cc:654] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.346262 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:14.387588 20823 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.389055 20823 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.392297 20823 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.393504 20797 tablet_bootstrap.cc:654] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.395746 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.398519 20820 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.399237 20820 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.400467 20869 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:14.397823 20506 catalog_manager.cc:5526] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:14.399267 20799 tablet_bootstrap.cc:492] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.403666 20799 ts_tablet_manager.cc:1397] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.096s user 0.033s sys 0.021s
I20250116 02:58:14.397775 20824 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.407078 20799 raft_consensus.cc:357] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.408213 20799 raft_consensus.cc:383] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.408586 20799 raft_consensus.cc:738] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.409528 20799 consensus_queue.cc:260] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:14.410743 20649 leader_election.cc:343] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.417438 20799 ts_tablet_manager.cc:1428] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250116 02:58:14.418128 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:14.418740 20799 tablet_bootstrap.cc:492] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:14.426855 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:14.430493 20810 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.423556 20503 catalog_manager.cc:5526] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:14.431727 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.426828 20607 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.433642 20756 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:14.439313 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:14.439446 20572 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:14.440739 20820 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.441499 20862 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:14.453218 20799 tablet_bootstrap.cc:654] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
W20250116 02:58:14.448244 20647 leader_election.cc:343] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.457352 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:14.446825 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:14.457363 20647 leader_election.cc:343] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.459045 20647 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:14.459839 20810 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:14.461620 20572 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:58:14.463158 20647 leader_election.cc:343] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.448338 20824 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.463951 20820 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:58:14.461791 20607 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:14.462091 20862 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:14.468171 20649 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:14.468511 20862 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.471681 20823 consensus_queue.cc:1035] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.462491 20804 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.471077 20810 raft_consensus.cc:2798] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.474965 20810 raft_consensus.cc:695] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:14.475898 20810 consensus_queue.cc:237] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.476778 20804 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:14.477730 20798 tablet_bootstrap.cc:492] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:14.478168 20798 ts_tablet_manager.cc:1397] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.177s user 0.029s sys 0.010s
I20250116 02:58:14.478315 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:14.479431 20683 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.481232 20798 raft_consensus.cc:357] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:14.479611 20649 consensus_peers.cc:487] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb -> Peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Couldn't send request to peer f65e8d7f76144fab9725ca32a6328c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:14.482594 20798 raft_consensus.cc:383] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.483187 20798 raft_consensus.cc:738] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.483961 20798 consensus_queue.cc:260] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.488885 20798 ts_tablet_manager.cc:1428] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20250116 02:58:14.487681 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:14.481197 20824 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20250116 02:58:14.490147 20798 tablet_bootstrap.cc:492] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:14.504477 20797 tablet_bootstrap.cc:492] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:14.505411 20797 ts_tablet_manager.cc:1397] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.210s user 0.014s sys 0.004s
I20250116 02:58:14.506793 20683 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:14.507956 20720 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:14.500787 20862 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.508682 20797 raft_consensus.cc:357] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.509112 20804 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.509646 20797 raft_consensus.cc:383] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.510035 20797 raft_consensus.cc:738] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.510716 20804 raft_consensus.cc:695] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:14.511227 20797 consensus_queue.cc:260] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.490865 20606 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.511698 20804 consensus_queue.cc:237] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.513378 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:14.514199 20683 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.519132 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:14.519986 20756 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.524982 20797 ts_tablet_manager.cc:1428] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250116 02:58:14.526119 20797 tablet_bootstrap.cc:492] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:14.529964 20756 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:14.536715 20572 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:14.537652 20820 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.538041 20863 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.538564 20863 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.541106 20863 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.543236 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.544649 20606 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
W20250116 02:58:14.545231 20571 leader_election.cc:343] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.547657 20798 tablet_bootstrap.cc:654] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.549626 20850 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:14.550802 20797 tablet_bootstrap.cc:654] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.556255 20799 tablet_bootstrap.cc:492] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.556864 20799 ts_tablet_manager.cc:1397] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.138s user 0.009s sys 0.002s
I20250116 02:58:14.551484 20862 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.567973 20503 catalog_manager.cc:5526] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:14.560002 20804 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.575802 20799 raft_consensus.cc:357] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.577162 20799 raft_consensus.cc:383] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.577539 20799 raft_consensus.cc:738] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.578564 20799 consensus_queue.cc:260] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.586050 20799 ts_tablet_manager.cc:1428] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.029s user 0.009s sys 0.002s
I20250116 02:58:14.587767 20799 tablet_bootstrap.cc:492] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:14.590744 20683 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:14.584389 20804 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.605939 20850 consensus_queue.cc:1035] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.606827 20506 catalog_manager.cc:5526] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:14.579864 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:14.612233 20820 raft_consensus.cc:695] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:14.613567 20820 consensus_queue.cc:237] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:14.619912 20572 leader_election.cc:343] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.620980 20572 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:14.621984 20862 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:14.628588 20862 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.633080 20862 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.634842 20798 tablet_bootstrap.cc:492] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:14.635334 20862 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:14.635435 20798 ts_tablet_manager.cc:1397] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.146s user 0.017s sys 0.007s
I20250116 02:58:14.639804 20799 tablet_bootstrap.cc:654] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.643487 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:14.644320 20804 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:58:14.645129 20572 leader_election.cc:343] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.645586 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.646183 20682 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:14.642433 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.646540 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:14.647850 20571 leader_election.cc:343] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:14.648030 20723 leader_election.cc:343] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:14.648347 20571 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:14.649196 20887 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:14.650090 20720 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:14.650404 20887 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.650679 20798 raft_consensus.cc:357] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.651507 20804 raft_consensus.cc:2798] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:14.652753 20804 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:14.653220 20804 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.651508 20798 raft_consensus.cc:383] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.656962 20798 raft_consensus.cc:738] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.661839 20869 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.662600 20869 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.665891 20869 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:14.666262 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.667846 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.668781 20683 raft_consensus.cc:2388] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:58:14.671026 20883 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.673749 20720 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:14.675789 20868 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.688764 20799 tablet_bootstrap.cc:492] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.689456 20799 ts_tablet_manager.cc:1397] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.102s user 0.012s sys 0.001s
I20250116 02:58:14.692921 20797 tablet_bootstrap.cc:492] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:14.693071 20799 raft_consensus.cc:357] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.694923 20799 raft_consensus.cc:383] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.695255 20799 raft_consensus.cc:738] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.696137 20799 consensus_queue.cc:260] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:14.697127 20720 consensus_peers.cc:487] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f -> Peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Couldn't send request to peer 231d91433a404468a44ea0a107345aeb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:14.658996 20798 consensus_queue.cc:260] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.699414 20799 ts_tablet_manager.cc:1428] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20250116 02:58:14.700654 20799 tablet_bootstrap.cc:492] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:14.706522 20798 ts_tablet_manager.cc:1428] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.070s user 0.006s sys 0.000s
I20250116 02:58:14.716508 20798 tablet_bootstrap.cc:492] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:14.719110 20506 catalog_manager.cc:5526] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:14.708487 20868 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:14.718343 20797 ts_tablet_manager.cc:1397] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.192s user 0.028s sys 0.007s
I20250116 02:58:14.725462 20799 tablet_bootstrap.cc:654] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.736447 20804 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.738085 20797 raft_consensus.cc:357] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.739300 20797 raft_consensus.cc:383] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.739667 20797 raft_consensus.cc:738] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.741649 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:14.742331 20683 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.742660 20798 tablet_bootstrap.cc:654] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.745680 20804 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:58:14.747753 20571 consensus_peers.cc:487] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Couldn't send request to peer 231d91433a404468a44ea0a107345aeb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:14.750267 20804 consensus_queue.cc:1035] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.753070 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:14.740891 20797 consensus_queue.cc:260] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.764148 20607 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.774943 20607 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:14.762373 20683 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:14.778597 20720 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:14.779719 20868 raft_consensus.cc:2798] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:14.780364 20868 raft_consensus.cc:695] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:14.781319 20868 consensus_queue.cc:237] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.779371 20799 tablet_bootstrap.cc:492] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.784449 20799 ts_tablet_manager.cc:1397] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.084s user 0.015s sys 0.003s
I20250116 02:58:14.785878 20797 ts_tablet_manager.cc:1428] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.063s user 0.008s sys 0.001s
I20250116 02:58:14.792775 20862 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.793344 20799 raft_consensus.cc:357] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.794294 20799 raft_consensus.cc:383] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.794629 20799 raft_consensus.cc:738] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.796123 20799 consensus_queue.cc:260] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.799248 20799 ts_tablet_manager.cc:1428] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20250116 02:58:14.800395 20799 tablet_bootstrap.cc:492] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:14.787356 20797 tablet_bootstrap.cc:492] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:14.813405 20810 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.816318 20506 catalog_manager.cc:5526] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:14.825261 20868 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:14.829905 20868 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.832232 20868 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:14.835934 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:14.836822 20605 raft_consensus.cc:2388] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:14.837460 20887 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.837770 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:14.839077 20798 tablet_bootstrap.cc:492] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:14.839576 20798 ts_tablet_manager.cc:1397] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.123s user 0.030s sys 0.056s
I20250116 02:58:14.841305 20720 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:14.842815 20799 tablet_bootstrap.cc:654] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.843137 20804 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:14.852908 20798 raft_consensus.cc:357] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.853945 20798 raft_consensus.cc:383] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.854290 20798 raft_consensus.cc:738] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.855062 20798 consensus_queue.cc:260] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.857533 20798 ts_tablet_manager.cc:1428] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.018s user 0.003s sys 0.004s
I20250116 02:58:14.860543 20798 tablet_bootstrap.cc:492] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:14.863067 20797 tablet_bootstrap.cc:654] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.858428 20804 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:14.866533 20810 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.880904 20798 tablet_bootstrap.cc:654] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.903328 20799 tablet_bootstrap.cc:492] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.903878 20799 ts_tablet_manager.cc:1397] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.104s user 0.023s sys 0.002s
I20250116 02:58:14.917897 20799 raft_consensus.cc:357] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.919055 20799 raft_consensus.cc:383] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.919310 20799 raft_consensus.cc:738] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.919862 20799 consensus_queue.cc:260] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.922667 20799 ts_tablet_manager.cc:1428] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250116 02:58:14.923712 20799 tablet_bootstrap.cc:492] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:14.932749 20810 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.933866 20798 tablet_bootstrap.cc:492] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:14.932723 20799 tablet_bootstrap.cc:654] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.934338 20798 ts_tablet_manager.cc:1397] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.024s
I20250116 02:58:14.933125 20797 tablet_bootstrap.cc:492] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:14.934962 20797 ts_tablet_manager.cc:1397] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.148s user 0.016s sys 0.018s
I20250116 02:58:14.937269 20798 raft_consensus.cc:357] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.937507 20797 raft_consensus.cc:357] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.938215 20798 raft_consensus.cc:383] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.938321 20797 raft_consensus.cc:383] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.938557 20798 raft_consensus.cc:738] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.939690 20798 consensus_queue.cc:260] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.938711 20797 raft_consensus.cc:738] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.942392 20798 ts_tablet_manager.cc:1428] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250116 02:58:14.942292 20797 consensus_queue.cc:260] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.945025 20797 ts_tablet_manager.cc:1428] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250116 02:58:14.946373 20798 tablet_bootstrap.cc:492] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:14.956688 20798 tablet_bootstrap.cc:654] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.946530 20797 tablet_bootstrap.cc:492] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:14.948911 20799 tablet_bootstrap.cc:492] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:14.960922 20799 ts_tablet_manager.cc:1397] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.003s
I20250116 02:58:14.964036 20799 raft_consensus.cc:357] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.969846 20797 tablet_bootstrap.cc:654] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:14.970134 20882 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.964885 20799 raft_consensus.cc:383] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.972479 20799 raft_consensus.cc:738] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.979811 20799 consensus_queue.cc:260] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.980777 20804 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:14.982189 20799 ts_tablet_manager.cc:1428] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250116 02:58:14.989671 20797 tablet_bootstrap.cc:492] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:14.990278 20797 ts_tablet_manager.cc:1397] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.002s
I20250116 02:58:14.995222 20799 tablet_bootstrap.cc:492] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:14.997184 20797 raft_consensus.cc:357] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:14.998046 20797 raft_consensus.cc:383] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:14.998370 20797 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:14.999138 20797 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.003077 20799 tablet_bootstrap.cc:654] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.007401 20797 ts_tablet_manager.cc:1428] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250116 02:58:15.007542 20605 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.008579 20797 tablet_bootstrap.cc:492] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:15.014853 20804 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.015515 20804 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.016994 20798 tablet_bootstrap.cc:492] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.017570 20798 ts_tablet_manager.cc:1397] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.071s user 0.018s sys 0.010s
I20250116 02:58:15.020084 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.020677 20798 raft_consensus.cc:357] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.021431 20798 raft_consensus.cc:383] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.021728 20798 raft_consensus.cc:738] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
W20250116 02:58:15.021843 20723 leader_election.cc:343] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.022471 20804 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:15.022485 20798 consensus_queue.cc:260] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.023808 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.024667 20683 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:15.025092 20798 ts_tablet_manager.cc:1428] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250116 02:58:15.026207 20720 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.027709 20896 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.028148 20896 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.028548 20896 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.031443 20798 tablet_bootstrap.cc:492] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.036934 20797 tablet_bootstrap.cc:654] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.042918 20895 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:15.044867 20798 tablet_bootstrap.cc:654] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.066327 20896 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.070365 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:58:15.072343 20723 leader_election.cc:343] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.076673 20799 tablet_bootstrap.cc:492] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:15.078958 20799 ts_tablet_manager.cc:1397] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.084s user 0.045s sys 0.033s
I20250116 02:58:15.079692 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:15.080437 20682 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.081869 20799 raft_consensus.cc:357] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.082839 20799 raft_consensus.cc:383] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.083302 20799 raft_consensus.cc:738] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.084108 20799 consensus_queue.cc:260] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.086750 20896 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:15.088379 20799 ts_tablet_manager.cc:1428] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250116 02:58:15.089882 20799 tablet_bootstrap.cc:492] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:15.089978 20798 tablet_bootstrap.cc:492] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.090590 20798 ts_tablet_manager.cc:1397] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.009s
I20250116 02:58:15.094568 20797 tablet_bootstrap.cc:492] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:15.095163 20797 ts_tablet_manager.cc:1397] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.087s user 0.026s sys 0.008s
I20250116 02:58:15.096808 20799 tablet_bootstrap.cc:654] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.099282 20682 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:15.100623 20720 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.101877 20896 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.102408 20896 raft_consensus.cc:695] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:15.103348 20896 consensus_queue.cc:237] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.100761 20798 raft_consensus.cc:357] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.104611 20798 raft_consensus.cc:383] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.104879 20798 raft_consensus.cc:738] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.105571 20798 consensus_queue.cc:260] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.108037 20798 ts_tablet_manager.cc:1428] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250116 02:58:15.109056 20798 tablet_bootstrap.cc:492] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.109264 20797 raft_consensus.cc:357] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.110193 20797 raft_consensus.cc:383] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.110483 20797 raft_consensus.cc:738] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.114316 20797 consensus_queue.cc:260] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.116618 20798 tablet_bootstrap.cc:654] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.117624 20797 ts_tablet_manager.cc:1428] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250116 02:58:15.118402 20804 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.118942 20804 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.120859 20797 tablet_bootstrap.cc:492] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:15.122380 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.124176 20804 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:58:15.125877 20723 leader_election.cc:343] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.132072 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.133077 20682 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:15.134845 20720 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.141057 20868 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.141589 20868 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.141964 20868 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.143028 20797 tablet_bootstrap.cc:654] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.151437 20895 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.151229 20887 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:15.152135 20895 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.157861 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.155854 20895 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:58:15.159588 20723 leader_election.cc:343] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.163247 20683 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.175577 20868 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.176709 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.180130 20895 consensus_queue.cc:1035] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:15.183091 20681 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:15.184650 20720 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.185680 20895 raft_consensus.cc:2798] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.186141 20895 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.186514 20895 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.193625 20868 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:15.196074 20797 tablet_bootstrap.cc:492] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:15.196628 20797 ts_tablet_manager.cc:1397] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.004s
I20250116 02:58:15.196913 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:15.197734 20680 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.194813 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:15.200542 20798 tablet_bootstrap.cc:492] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
W20250116 02:58:15.200910 20723 leader_election.cc:343] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.201117 20798 ts_tablet_manager.cc:1397] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.092s user 0.033s sys 0.049s
I20250116 02:58:15.204128 20798 raft_consensus.cc:357] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.204946 20798 raft_consensus.cc:383] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.205299 20798 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.206090 20798 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.207717 20887 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:15.208920 20798 ts_tablet_manager.cc:1428] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250116 02:58:15.210004 20798 tablet_bootstrap.cc:492] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.204784 20863 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.215227 20863 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.216514 20797 raft_consensus.cc:357] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.217299 20797 raft_consensus.cc:383] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.217572 20797 raft_consensus.cc:738] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.219197 20896 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.218577 20797 consensus_queue.cc:260] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.218777 20503 catalog_manager.cc:5526] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:15.219725 20896 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.221966 20896 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:15.223012 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.223822 20605 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:15.224467 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.224889 20798 tablet_bootstrap.cc:654] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.225070 20723 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:58:15.228039 20720 leader_election.cc:343] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.233580 20799 tablet_bootstrap.cc:492] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:15.234153 20799 ts_tablet_manager.cc:1397] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.144s user 0.015s sys 0.011s
I20250116 02:58:15.235999 20810 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.236544 20810 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.237722 20797 ts_tablet_manager.cc:1428] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.041s user 0.002s sys 0.007s
I20250116 02:58:15.237668 20799 raft_consensus.cc:357] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.238577 20799 raft_consensus.cc:383] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.238952 20799 raft_consensus.cc:738] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.239930 20863 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.240175 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.230792 20896 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.241006 20896 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.241420 20896 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.241586 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.239655 20799 consensus_queue.cc:260] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:15.242287 20926 log.cc:926] Time spent T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
W20250116 02:58:15.243507 20571 leader_election.cc:343] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.244513 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.246134 20810 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:58:15.247460 20572 leader_election.cc:343] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.247967 20572 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:15.249982 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.252231 20863 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:15.254400 20647 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.255709 20880 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:15.256186 20799 ts_tablet_manager.cc:1428] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.022s user 0.002s sys 0.007s
I20250116 02:58:15.254036 20797 tablet_bootstrap.cc:492] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:15.257148 20799 tablet_bootstrap.cc:492] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:15.262105 20804 consensus_queue.cc:1035] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:15.265794 20896 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.255823 20680 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:15.272943 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:15.273128 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:15.276101 20607 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.276402 20896 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:15.261209 20895 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.277182 20720 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:58:15.279173 20720 leader_election.cc:343] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.280352 20797 tablet_bootstrap.cc:654] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.283728 20896 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.285008 20896 raft_consensus.cc:695] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:15.285962 20896 consensus_queue.cc:237] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.289069 20882 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:15.289696 20607 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:15.291016 20723 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:15.292140 20804 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.292598 20804 raft_consensus.cc:695] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:15.293423 20804 consensus_queue.cc:237] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.289781 20882 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.315819 20863 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.316849 20882 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.316478 20863 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.368639 20798 tablet_bootstrap.cc:492] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.364580 20862 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.357507 20882 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.371290 20798 ts_tablet_manager.cc:1397] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.162s user 0.016s sys 0.012s
I20250116 02:58:15.371649 20882 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.372606 20862 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.374159 20798 raft_consensus.cc:357] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.374843 20798 raft_consensus.cc:383] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.375113 20798 raft_consensus.cc:738] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.375409 20862 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.326416 20895 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:15.318106 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:15.375981 20798 consensus_queue.cc:260] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.319720 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.318986 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:15.382282 20799 tablet_bootstrap.cc:654] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.324393 20680 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.392402 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.326421 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.376744 20681 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.383343 20882 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.383766 20606 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.395268 20647 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.396832 20882 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:58:15.384425 20863 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.384685 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:15.400576 20647 leader_election.cc:343] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.385520 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.386018 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.386232 20798 ts_tablet_manager.cc:1428] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20250116 02:58:15.386940 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:58:15.404219 20572 leader_election.cc:343] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.404801 20682 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
W20250116 02:58:15.371038 20925 log.cc:926] Time spent T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.055s user 0.002s sys 0.000s
I20250116 02:58:15.405606 20797 tablet_bootstrap.cc:492] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:15.387893 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.406241 20797 ts_tablet_manager.cc:1397] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.152s user 0.024s sys 0.020s
I20250116 02:58:15.406535 20755 raft_consensus.cc:2388] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:58:15.406932 20571 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:15.404266 20798 tablet_bootstrap.cc:492] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.408125 20572 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:15.409180 20797 raft_consensus.cc:357] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.409978 20797 raft_consensus.cc:383] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.410280 20797 raft_consensus.cc:738] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.411022 20797 consensus_queue.cc:260] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.412813 20797 ts_tablet_manager.cc:1428] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250116 02:58:15.413779 20797 tablet_bootstrap.cc:492] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:15.408180 20863 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.415030 20863 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.415383 20863 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.417285 20804 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.417959 20804 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.421289 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.422197 20804 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:15.421919 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:58:15.423156 20723 leader_election.cc:343] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:15.423664 20720 leader_election.cc:343] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.424117 20720 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.425400 20804 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:15.425107 20863 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.426908 20605 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:15.428097 20797 tablet_bootstrap.cc:654] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.428185 20649 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:15.428851 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:15.429137 20882 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.429484 20682 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.429646 20882 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.430137 20882 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.430251 20887 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.430927 20887 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.433367 20887 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.434278 20863 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.434449 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.434667 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.435390 20683 raft_consensus.cc:2388] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:58:15.436931 20572 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:15.438421 20863 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.430964 20798 tablet_bootstrap.cc:654] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.439890 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:58:15.441992 20572 leader_election.cc:343] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.406543 20606 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:15.451401 20862 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:15.453085 20723 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:15.454057 20896 raft_consensus.cc:2798] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.454856 20896 raft_consensus.cc:695] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:15.455763 20896 consensus_queue.cc:237] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:15.459930 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:15.465662 20681 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:15.473510 20863 raft_consensus.cc:2743] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:15.477488 20503 catalog_manager.cc:5526] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:15.479641 20503 catalog_manager.cc:5526] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:15.481511 20682 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:15.480095 20882 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.483119 20571 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:15.485961 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:15.486784 20606 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.494462 20896 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.495268 20896 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.498512 20896 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:15.499099 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.500717 20863 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.501435 20863 raft_consensus.cc:695] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:15.502482 20863 consensus_queue.cc:237] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:15.507875 20723 leader_election.cc:343] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.511255 20798 tablet_bootstrap.cc:492] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.511839 20798 ts_tablet_manager.cc:1397] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.108s user 0.018s sys 0.016s
I20250116 02:58:15.511617 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.514590 20798 raft_consensus.cc:357] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:15.515067 20720 leader_election.cc:343] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.515650 20720 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.516448 20606 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:15.516997 20896 raft_consensus.cc:2743] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:15.517827 20649 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:15.517889 20798 raft_consensus.cc:383] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.519084 20798 raft_consensus.cc:738] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.519764 20798 consensus_queue.cc:260] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.520234 20880 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.522251 20798 ts_tablet_manager.cc:1428] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250116 02:58:15.522047 20506 catalog_manager.cc:5526] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:15.523289 20798 tablet_bootstrap.cc:492] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.526429 20503 catalog_manager.cc:5526] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:15.531023 20798 tablet_bootstrap.cc:654] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.531720 20797 tablet_bootstrap.cc:492] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:15.540051 20797 ts_tablet_manager.cc:1397] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.126s user 0.012s sys 0.006s
I20250116 02:58:15.532531 20882 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.533546 20880 raft_consensus.cc:695] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:15.543995 20880 consensus_queue.cc:237] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.544989 20798 tablet_bootstrap.cc:492] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.545532 20798 ts_tablet_manager.cc:1397] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.000s
I20250116 02:58:15.548401 20798 raft_consensus.cc:357] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.549237 20798 raft_consensus.cc:383] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.531013 20799 tablet_bootstrap.cc:492] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:15.541263 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:15.549522 20798 raft_consensus.cc:738] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.551554 20799 ts_tablet_manager.cc:1397] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.295s user 0.025s sys 0.020s
W20250116 02:58:15.552213 20647 leader_election.cc:343] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.552443 20798 consensus_queue.cc:260] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.554646 20799 raft_consensus.cc:357] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.555430 20798 ts_tablet_manager.cc:1428] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20250116 02:58:15.555796 20797 raft_consensus.cc:357] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.556604 20798 tablet_bootstrap.cc:492] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
W20250116 02:58:15.557520 20723 consensus_peers.cc:487] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f -> Peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Couldn't send request to peer f65e8d7f76144fab9725ca32a6328c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:15.555486 20799 raft_consensus.cc:383] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.558670 20799 raft_consensus.cc:738] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.559381 20799 consensus_queue.cc:260] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.561887 20799 ts_tablet_manager.cc:1428] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20250116 02:58:15.556560 20797 raft_consensus.cc:383] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.563139 20798 tablet_bootstrap.cc:654] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.563953 20799 tablet_bootstrap.cc:492] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:15.563125 20797 raft_consensus.cc:738] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.567373 20797 consensus_queue.cc:260] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.571404 20799 tablet_bootstrap.cc:654] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.573793 20797 ts_tablet_manager.cc:1428] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20250116 02:58:15.575076 20797 tablet_bootstrap.cc:492] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:15.575614 20503 catalog_manager.cc:5526] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:15.582183 20797 tablet_bootstrap.cc:654] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.594580 20799 tablet_bootstrap.cc:492] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:15.595146 20799 ts_tablet_manager.cc:1397] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.001s
I20250116 02:58:15.597790 20799 raft_consensus.cc:357] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.598579 20799 raft_consensus.cc:383] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.598937 20799 raft_consensus.cc:738] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.599869 20863 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.600499 20863 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.604370 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.604701 20863 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.599608 20799 consensus_queue.cc:260] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.605535 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:58:15.606994 20571 leader_election.cc:343] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.607098 20755 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:15.607877 20799 ts_tablet_manager.cc:1428] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20250116 02:58:15.608651 20798 tablet_bootstrap.cc:492] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.609164 20798 ts_tablet_manager.cc:1397] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.001s
I20250116 02:58:15.609683 20799 tablet_bootstrap.cc:492] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:15.610463 20572 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:15.612396 20798 raft_consensus.cc:357] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.613242 20798 raft_consensus.cc:383] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.613765 20798 raft_consensus.cc:738] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.614845 20798 consensus_queue.cc:260] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.616184 20863 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.616624 20863 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.616997 20863 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.617563 20797 tablet_bootstrap.cc:492] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:15.617794 20798 ts_tablet_manager.cc:1428] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20250116 02:58:15.618235 20797 ts_tablet_manager.cc:1397] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.002s
I20250116 02:58:15.619431 20798 tablet_bootstrap.cc:492] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.626675 20799 tablet_bootstrap.cc:654] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.625128 20863 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.629401 20798 tablet_bootstrap.cc:654] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.621788 20797 raft_consensus.cc:357] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.629905 20797 raft_consensus.cc:383] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.630224 20797 raft_consensus.cc:738] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.630954 20797 consensus_queue.cc:260] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.631448 20939 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.632607 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:15.633307 20863 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.633224 20804 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:15.635260 20863 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.635892 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:15.635838 20863 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.636624 20755 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.638191 20797 ts_tablet_manager.cc:1428] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.020s user 0.003s sys 0.004s
I20250116 02:58:15.639981 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.644377 20682 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:15.640153 20863 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.640369 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:15.640282 20571 leader_election.cc:343] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.647341 20571 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:15.632328 20939 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:15.647925 20572 leader_election.cc:343] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.649027 20797 tablet_bootstrap.cc:492] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:15.642987 20798 tablet_bootstrap.cc:492] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.649606 20939 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.650601 20798 ts_tablet_manager.cc:1397] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.000s
I20250116 02:58:15.650813 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.651856 20606 raft_consensus.cc:2388] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:58:15.653865 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.655705 20647 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.656317 20797 tablet_bootstrap.cc:654] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.656564 20882 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.649161 20863 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.657891 20863 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.658468 20863 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.653681 20798 raft_consensus.cc:357] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.659621 20798 raft_consensus.cc:383] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.659937 20798 raft_consensus.cc:738] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.660498 20799 tablet_bootstrap.cc:492] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:15.661037 20799 ts_tablet_manager.cc:1397] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.000s
I20250116 02:58:15.660637 20798 consensus_queue.cc:260] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.664419 20798 ts_tablet_manager.cc:1428] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250116 02:58:15.668926 20798 tablet_bootstrap.cc:492] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.670176 20755 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:15.671685 20572 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:15.673270 20862 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.673813 20862 raft_consensus.cc:695] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:15.673976 20882 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:15.674839 20862 consensus_queue.cc:237] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.677350 20799 raft_consensus.cc:357] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.678274 20799 raft_consensus.cc:383] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.678551 20799 raft_consensus.cc:738] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.679343 20799 consensus_queue.cc:260] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.670226 20863 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.681669 20799 ts_tablet_manager.cc:1428] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250116 02:58:15.684690 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:15.685475 20682 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.685003 20799 tablet_bootstrap.cc:492] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:15.689597 20863 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.696058 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:58:15.700621 20572 leader_election.cc:343] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.705818 20797 tablet_bootstrap.cc:492] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:15.706420 20797 ts_tablet_manager.cc:1397] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.008s
I20250116 02:58:15.713505 20682 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:15.714798 20571 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:15.716060 20862 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.716544 20862 raft_consensus.cc:695] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:15.716635 20798 tablet_bootstrap.cc:654] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.717408 20862 consensus_queue.cc:237] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.724920 20797 raft_consensus.cc:357] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.725756 20797 raft_consensus.cc:383] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.726027 20797 raft_consensus.cc:738] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.727823 20797 consensus_queue.cc:260] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.730705 20797 ts_tablet_manager.cc:1428] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250116 02:58:15.731175 20882 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.731603 20797 tablet_bootstrap.cc:492] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:15.731910 20882 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.734911 20798 tablet_bootstrap.cc:492] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.735352 20798 ts_tablet_manager.cc:1397] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.067s user 0.018s sys 0.000s
I20250116 02:58:15.735791 20503 catalog_manager.cc:5526] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:15.737882 20798 raft_consensus.cc:357] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.738590 20798 raft_consensus.cc:383] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.738961 20797 tablet_bootstrap.cc:654] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.738888 20798 raft_consensus.cc:738] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.739820 20798 consensus_queue.cc:260] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.742551 20798 ts_tablet_manager.cc:1428] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250116 02:58:15.743095 20804 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:15.744130 20798 tablet_bootstrap.cc:492] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.752328 20798 tablet_bootstrap.cc:654] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.752285 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:15.754196 20649 leader_election.cc:343] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.754456 20797 tablet_bootstrap.cc:492] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:15.755028 20797 ts_tablet_manager.cc:1397] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.005s
I20250116 02:58:15.757107 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.758150 20754 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:15.759642 20797 raft_consensus.cc:357] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.760581 20797 raft_consensus.cc:383] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.760874 20797 raft_consensus.cc:738] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.761798 20797 consensus_queue.cc:260] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.764389 20797 ts_tablet_manager.cc:1428] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250116 02:58:15.764830 20798 tablet_bootstrap.cc:492] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.765357 20798 ts_tablet_manager.cc:1397] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.006s
I20250116 02:58:15.765439 20797 tablet_bootstrap.cc:492] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:15.768739 20798 raft_consensus.cc:357] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.769583 20798 raft_consensus.cc:383] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.769866 20798 raft_consensus.cc:738] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.764873 20799 tablet_bootstrap.cc:654] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.771162 20882 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.772296 20647 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.772969 20797 tablet_bootstrap.cc:654] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.770541 20798 consensus_queue.cc:260] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.773651 20882 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.774070 20882 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.774423 20882 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.776909 20798 ts_tablet_manager.cc:1428] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20250116 02:58:15.778252 20798 tablet_bootstrap.cc:492] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.785209 20798 tablet_bootstrap.cc:654] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.785750 20797 tablet_bootstrap.cc:492] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:15.786322 20797 ts_tablet_manager.cc:1397] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.007s
I20250116 02:58:15.788005 20880 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.788533 20880 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.791288 20799 tablet_bootstrap.cc:492] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:15.791798 20799 ts_tablet_manager.cc:1397] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.010s
I20250116 02:58:15.791893 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.792619 20606 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:15.793859 20649 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:15.794834 20939 raft_consensus.cc:2798] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.790277 20797 raft_consensus.cc:357] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.795516 20939 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.795848 20797 raft_consensus.cc:383] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.796044 20939 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.796240 20797 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.796917 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.797081 20797 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.798451 20880 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.799444 20797 ts_tablet_manager.cc:1428] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250116 02:58:15.803486 20882 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.804883 20797 tablet_bootstrap.cc:492] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:15.805331 20798 tablet_bootstrap.cc:492] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:15.805835 20798 ts_tablet_manager.cc:1397] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.000s
I20250116 02:58:15.807014 20882 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.807758 20862 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.808260 20862 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.809063 20798 raft_consensus.cc:357] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.809931 20798 raft_consensus.cc:383] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.810185 20798 raft_consensus.cc:738] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.794834 20799 raft_consensus.cc:357] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.810362 20862 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.802567 20939 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.810770 20799 raft_consensus.cc:383] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:15.811085 20799 raft_consensus.cc:738] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:15.811657 20799 consensus_queue.cc:260] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.812211 20939 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.812858 20798 consensus_queue.cc:260] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.813987 20799 ts_tablet_manager.cc:1428] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20250116 02:58:15.813774 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:15.814481 20606 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.818773 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.797729 20754 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
W20250116 02:58:15.820087 20571 leader_election.cc:343] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.821715 20606 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:15.822551 20939 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.823452 20649 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:15.821933 20798 ts_tablet_manager.cc:1428] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250116 02:58:15.823112 20939 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.824350 20797 tablet_bootstrap.cc:654] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.824846 20798 tablet_bootstrap.cc:492] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:15.827817 20939 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.829525 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.829779 20896 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250116 02:58:15.831348 20649 leader_election.cc:343] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.832274 20939 raft_consensus.cc:2798] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.833307 20939 raft_consensus.cc:695] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:15.834102 20810 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.834152 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:15.834256 20939 consensus_queue.cc:237] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.834316 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.836046 20810 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.834447 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:15.837808 20756 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.838264 20810 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.835718 20755 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.842883 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.824846 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:15.848213 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:15.849854 20649 leader_election.cc:343] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:15.850876 20649 leader_election.cc:343] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.853446 20799 tablet_bootstrap.cc:492] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
W20250116 02:58:15.859169 20647 leader_election.cc:343] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.859871 20647 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.860864 20880 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250116 02:58:15.879014 20572 leader_election.cc:343] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.879781 20572 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:15.880932 20862 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:15.897660 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:15.898873 20754 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:15.912488 20647 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.913705 20882 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:15.914161 20882 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:15.914486 20882 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.912626 20896 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.916070 20896 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.923233 20810 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:15.923974 20810 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.927626 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.927836 20810 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:58:15.929415 20649 leader_election.cc:343] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.908070 20505 catalog_manager.cc:5526] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:15.935441 20756 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:15.936305 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:15.935743 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 56823.2us
I20250116 02:58:15.937486 20896 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:15.947145 20755 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:15.947615 20896 consensus_queue.cc:1035] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250116 02:58:15.942993 20647 leader_election.cc:343] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:15.948879 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:15.949895 20682 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:58:15.949998 20804 consensus_queue.cc:1035] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:15.948932 20647 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.953018 20647 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:15.953931 20880 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:15.954116 20896 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:15.954703 20880 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:15.955253 20880 raft_consensus.cc:695] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:15.955653 20607 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.956220 20880 consensus_queue.cc:237] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.968789 20798 tablet_bootstrap.cc:654] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:15.943959 20882 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:15.993381 20882 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:15.985966 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:15.999972 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:16.000835 20882 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.001452 20882 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.000849 20755 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.003554 20804 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:16.000504 20723 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:16.012430 20804 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:15.998204 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:16.020594 20882 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.022918 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:16.023941 20756 raft_consensus.cc:2388] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 1: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:58:16.020658 20649 leader_election.cc:343] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.024890 20799 tablet_bootstrap.cc:654] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.028491 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:16.033691 20649 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:16.035702 20810 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.042310 20804 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:16.043725 20863 consensus_queue.cc:1035] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:16.044786 20755 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:16.046700 20756 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.062732 20797 tablet_bootstrap.cc:492] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:16.063342 20797 ts_tablet_manager.cc:1397] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.259s user 0.082s sys 0.074s
I20250116 02:58:16.057189 20647 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:16.075640 20798 tablet_bootstrap.cc:492] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:16.076166 20798 ts_tablet_manager.cc:1397] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.252s user 0.011s sys 0.007s
I20250116 02:58:16.079123 20798 raft_consensus.cc:357] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.079878 20798 raft_consensus.cc:383] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.080164 20798 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.079952 20504 catalog_manager.cc:5526] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:16.083225 20959 log.cc:926] Time spent T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250116 02:58:16.069891 20880 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:16.158057 20810 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:16.158390 20880 raft_consensus.cc:695] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:16.160123 20798 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.161263 20939 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.161829 20939 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.166126 20939 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.172394 20797 raft_consensus.cc:357] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.173182 20797 raft_consensus.cc:383] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.173434 20797 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.174005 20797 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.180176 20798 ts_tablet_manager.cc:1428] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.104s user 0.006s sys 0.003s
I20250116 02:58:16.181295 20798 tablet_bootstrap.cc:492] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
W20250116 02:58:16.184078 20572 consensus_peers.cc:487] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Couldn't send request to peer 17446c8dacf04461acb843547665250f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:16.187780 20797 ts_tablet_manager.cc:1428] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.124s user 0.010s sys 0.012s
I20250116 02:58:16.188642 20797 tablet_bootstrap.cc:492] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:16.173382 20880 consensus_queue.cc:237] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.200599 20868 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.201195 20868 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.211525 20863 consensus_queue.cc:1035] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:16.212654 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:16.216943 20797 tablet_bootstrap.cc:654] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.182909 20939 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20250116 02:58:16.215425 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:16.217835 20607 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
W20250116 02:58:16.227751 20968 log.cc:926] Time spent T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.060s user 0.003s sys 0.000s
I20250116 02:58:16.251863 20798 tablet_bootstrap.cc:654] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.258217 20649 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:58:16.261725 20965 log.cc:926] Time spent T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.099s user 0.001s sys 0.002s
I20250116 02:58:16.240916 20882 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20250116 02:58:16.263751 20882 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:16.264144 20882 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:16.264515 20882 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:16.262313 20647 leader_election.cc:343] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.264166 20799 tablet_bootstrap.cc:492] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:16.266660 20799 ts_tablet_manager.cc:1397] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.413s user 0.036s sys 0.015s
I20250116 02:58:16.267233 20797 tablet_bootstrap.cc:492] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:16.265794 20887 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:16.267727 20797 ts_tablet_manager.cc:1397] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.010s
I20250116 02:58:16.270756 20797 raft_consensus.cc:357] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.271615 20797 raft_consensus.cc:383] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.271885 20797 raft_consensus.cc:738] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.272866 20797 consensus_queue.cc:260] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.275171 20797 ts_tablet_manager.cc:1428] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20250116 02:58:16.281025 20887 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:16.281013 20868 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:16.282039 20882 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.283978 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:16.286074 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:16.286971 20607 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.289201 20797 tablet_bootstrap.cc:492] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:16.294700 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:16.300932 20797 tablet_bootstrap.cc:654] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.303308 20939 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:16.310184 20799 raft_consensus.cc:357] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.328087 20799 raft_consensus.cc:383] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.320583 20797 tablet_bootstrap.cc:492] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:16.330098 20799 raft_consensus.cc:738] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.317761 20504 catalog_manager.cc:5526] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:16.332171 20504 catalog_manager.cc:5526] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:16.314285 20882 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.330513 20797 ts_tablet_manager.cc:1397] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.001s
I20250116 02:58:16.321481 20720 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:16.338310 20804 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:58:16.338986 20607 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:16.341475 20649 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:16.343705 20939 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:16.344249 20939 raft_consensus.cc:695] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:16.344705 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:16.345184 20939 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.367776 20887 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.368489 20887 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.385831 20799 consensus_queue.cc:260] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.387950 20887 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.385886 20797 raft_consensus.cc:357] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.389400 20797 raft_consensus.cc:383] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.391024 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:16.394356 20681 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:16.394824 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:16.395643 20799 ts_tablet_manager.cc:1428] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.129s user 0.013s sys 0.012s
I20250116 02:58:16.394229 20797 raft_consensus.cc:738] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
W20250116 02:58:16.396472 20572 leader_election.cc:343] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.405995 20797 consensus_queue.cc:260] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.410076 20797 ts_tablet_manager.cc:1428] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.075s user 0.010s sys 0.008s
I20250116 02:58:16.411281 20797 tablet_bootstrap.cc:492] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
W20250116 02:58:16.414649 20647 leader_election.cc:343] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.417405 20882 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:16.407541 20571 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:16.407907 20799 tablet_bootstrap.cc:492] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:16.422132 20863 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:16.422698 20863 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:16.423087 20863 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.433387 20799 tablet_bootstrap.cc:654] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.434967 20887 consensus_queue.cc:1035] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:16.433444 20863 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.442492 20863 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.439240 20798 tablet_bootstrap.cc:492] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:16.443956 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:16.439970 20797 tablet_bootstrap.cc:654] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
W20250116 02:58:16.445683 20572 leader_election.cc:343] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.444296 20798 ts_tablet_manager.cc:1397] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.263s user 0.033s sys 0.030s
I20250116 02:58:16.450397 20798 raft_consensus.cc:357] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.452018 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:16.453439 20799 tablet_bootstrap.cc:492] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:16.454136 20798 raft_consensus.cc:383] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.454753 20798 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.454197 20682 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.454730 20799 ts_tablet_manager.cc:1397] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.002s
I20250116 02:58:16.461622 20799 raft_consensus.cc:357] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.462554 20799 raft_consensus.cc:383] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.463059 20799 raft_consensus.cc:738] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.470656 20682 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:16.481048 20571 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:16.482240 20863 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:16.482822 20863 raft_consensus.cc:695] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:16.483767 20863 consensus_queue.cc:237] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.470013 20799 consensus_queue.cc:260] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.472404 20798 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.489250 20880 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.489830 20880 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.493552 20798 ts_tablet_manager.cc:1428] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250116 02:58:16.494962 20798 tablet_bootstrap.cc:492] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:16.502941 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:16.503976 20607 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:16.505893 20880 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.506141 20649 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:16.508417 20799 ts_tablet_manager.cc:1428] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20250116 02:58:16.509557 20880 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:16.510478 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:16.510913 20880 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:16.511327 20880 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.511562 20799 tablet_bootstrap.cc:492] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:16.508917 20504 catalog_manager.cc:5526] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:16.512997 20798 tablet_bootstrap.cc:654] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.519412 20799 tablet_bootstrap.cc:654] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
W20250116 02:58:16.521236 20647 leader_election.cc:343] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.529593 20880 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.541234 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:16.541965 20607 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.546020 20939 consensus_queue.cc:1035] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250116 02:58:16.561661 20649 consensus_peers.cc:487] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb -> Peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Couldn't send request to peer f65e8d7f76144fab9725ca32a6328c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:16.550503 20607 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:16.564810 20880 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.570394 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:16.570827 20756 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.576416 20799 tablet_bootstrap.cc:492] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:16.577037 20799 ts_tablet_manager.cc:1397] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.066s user 0.016s sys 0.006s
W20250116 02:58:16.591042 20647 leader_election.cc:343] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.592865 20649 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:16.594208 20939 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:16.594792 20939 raft_consensus.cc:695] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:16.595677 20939 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.599577 20799 raft_consensus.cc:357] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.600445 20799 raft_consensus.cc:383] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.600725 20799 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.601400 20799 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.604579 20798 tablet_bootstrap.cc:492] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:16.605157 20798 ts_tablet_manager.cc:1397] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.110s user 0.034s sys 0.017s
I20250116 02:58:16.606267 20799 ts_tablet_manager.cc:1428] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20250116 02:58:16.607589 20799 tablet_bootstrap.cc:492] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:16.614708 20797 tablet_bootstrap.cc:492] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:16.615254 20797 ts_tablet_manager.cc:1397] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.204s user 0.056s sys 0.055s
I20250116 02:58:16.609302 20798 raft_consensus.cc:357] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.615891 20798 raft_consensus.cc:383] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.616179 20798 raft_consensus.cc:738] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.607633 20503 catalog_manager.cc:5526] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:16.622555 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:16.622761 20797 raft_consensus.cc:357] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:16.612605 20649 consensus_peers.cc:487] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb -> Peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Couldn't send request to peer f65e8d7f76144fab9725ca32a6328c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:16.624159 20798 consensus_queue.cc:260] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.626060 20797 raft_consensus.cc:383] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.626437 20797 raft_consensus.cc:738] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.627242 20797 consensus_queue.cc:260] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.631357 20797 ts_tablet_manager.cc:1428] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s
I20250116 02:58:16.631637 20798 ts_tablet_manager.cc:1428] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.026s user 0.004s sys 0.002s
I20250116 02:58:16.632550 20798 tablet_bootstrap.cc:492] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:16.632560 20797 tablet_bootstrap.cc:492] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:16.634902 20862 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.635565 20862 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.639554 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:16.640669 20799 tablet_bootstrap.cc:654] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
W20250116 02:58:16.641916 20571 leader_election.cc:343] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.637952 20862 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.643543 20797 tablet_bootstrap.cc:654] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.644101 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:16.646276 20572 leader_election.cc:343] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.646785 20572 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:16.647516 20862 raft_consensus.cc:2743] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:16.654047 20798 tablet_bootstrap.cc:654] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.670068 20799 tablet_bootstrap.cc:492] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:16.670661 20799 ts_tablet_manager.cc:1397] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.006s
I20250116 02:58:16.673518 20798 tablet_bootstrap.cc:492] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:16.674448 20798 ts_tablet_manager.cc:1397] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.005s
I20250116 02:58:16.677659 20798 raft_consensus.cc:357] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.678535 20798 raft_consensus.cc:383] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.678860 20798 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.679526 20798 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.682337 20880 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:16.684006 20799 raft_consensus.cc:357] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.684929 20799 raft_consensus.cc:383] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.685070 20797 tablet_bootstrap.cc:492] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:16.685281 20799 raft_consensus.cc:738] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.685618 20797 ts_tablet_manager.cc:1397] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.019s
I20250116 02:58:16.688439 20799 consensus_queue.cc:260] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.688808 20797 raft_consensus.cc:357] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.689625 20797 raft_consensus.cc:383] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.689929 20797 raft_consensus.cc:738] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.691004 20799 ts_tablet_manager.cc:1428] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250116 02:58:16.690732 20797 consensus_queue.cc:260] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.692207 20799 tablet_bootstrap.cc:492] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:16.700011 20799 tablet_bootstrap.cc:654] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.701864 20797 ts_tablet_manager.cc:1428] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250116 02:58:16.701773 20505 catalog_manager.cc:5526] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:16.712584 20797 tablet_bootstrap.cc:492] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:16.715041 20798 ts_tablet_manager.cc:1428] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20250116 02:58:16.716364 20798 tablet_bootstrap.cc:492] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:16.719821 20797 tablet_bootstrap.cc:654] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.729981 20798 tablet_bootstrap.cc:654] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.736903 20862 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.737509 20862 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.739776 20862 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.743458 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:16.747198 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:16.750182 20572 leader_election.cc:343] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:16.751681 20571 leader_election.cc:343] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.752692 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:16.754101 20862 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:16.754575 20869 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.755239 20869 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.756680 20798 tablet_bootstrap.cc:492] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:16.757529 20798 ts_tablet_manager.cc:1397] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.007s
I20250116 02:58:16.759756 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:16.760814 20607 raft_consensus.cc:2388] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:16.767153 20798 raft_consensus.cc:357] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.767848 20798 raft_consensus.cc:383] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.768126 20798 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.757746 20869 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:16.769688 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:16.771231 20720 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:16.772637 20868 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.768827 20798 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.783505 20868 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:16.786825 20798 ts_tablet_manager.cc:1428] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20250116 02:58:16.788089 20882 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.789505 20882 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.790377 20798 tablet_bootstrap.cc:492] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:16.791189 20797 tablet_bootstrap.cc:492] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:16.791757 20797 ts_tablet_manager.cc:1397] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.079s user 0.048s sys 0.020s
I20250116 02:58:16.794824 20797 raft_consensus.cc:357] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.795619 20797 raft_consensus.cc:383] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.795877 20797 raft_consensus.cc:738] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.796542 20797 consensus_queue.cc:260] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.797618 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:16.798691 20797 ts_tablet_manager.cc:1428] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250116 02:58:16.799413 20882 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.800237 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:16.800585 20649 leader_election.cc:343] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:16.801764 20647 leader_election.cc:343] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.802199 20647 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:16.803766 20798 tablet_bootstrap.cc:654] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.803992 20939 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:16.788936 20799 tablet_bootstrap.cc:492] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:16.805406 20799 ts_tablet_manager.cc:1397] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.113s user 0.029s sys 0.023s
I20250116 02:58:16.808461 20799 raft_consensus.cc:357] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.809249 20799 raft_consensus.cc:383] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.809507 20799 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.810297 20797 tablet_bootstrap.cc:492] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:16.811290 20799 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.813616 20799 ts_tablet_manager.cc:1428] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250116 02:58:16.815496 20798 tablet_bootstrap.cc:492] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:16.816464 20798 ts_tablet_manager.cc:1397] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.000s
I20250116 02:58:16.817816 20797 tablet_bootstrap.cc:654] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.819267 20798 raft_consensus.cc:357] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.820099 20798 raft_consensus.cc:383] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.820362 20798 raft_consensus.cc:738] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.820962 20798 consensus_queue.cc:260] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.827471 20798 ts_tablet_manager.cc:1428] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250116 02:58:16.828063 20799 tablet_bootstrap.cc:492] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:16.828529 20798 tablet_bootstrap.cc:492] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:16.835948 20799 tablet_bootstrap.cc:654] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.839215 20862 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.839545 20798 tablet_bootstrap.cc:654] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.839882 20862 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.843712 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:16.846249 20572 leader_election.cc:343] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.848073 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:16.849011 20681 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
W20250116 02:58:16.851052 20647 consensus_peers.cc:487] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb -> Peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Couldn't send request to peer 17446c8dacf04461acb843547665250f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:16.851604 20862 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.851792 20798 tablet_bootstrap.cc:492] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:16.852172 20799 tablet_bootstrap.cc:492] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:16.850979 20571 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:16.852491 20798 ts_tablet_manager.cc:1397] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.002s
I20250116 02:58:16.852620 20799 ts_tablet_manager.cc:1397] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.006s
I20250116 02:58:16.852990 20862 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:16.853484 20862 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:16.853921 20862 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.855358 20797 tablet_bootstrap.cc:492] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:16.855949 20797 ts_tablet_manager.cc:1397] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.046s user 0.016s sys 0.009s
I20250116 02:58:16.855455 20798 raft_consensus.cc:357] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.856299 20798 raft_consensus.cc:383] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.856271 20799 raft_consensus.cc:357] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.856781 20798 raft_consensus.cc:738] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.857195 20799 raft_consensus.cc:383] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.857532 20799 raft_consensus.cc:738] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.857540 20798 consensus_queue.cc:260] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.858875 20797 raft_consensus.cc:357] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.859562 20797 raft_consensus.cc:383] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.859819 20797 raft_consensus.cc:738] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.860374 20798 ts_tablet_manager.cc:1428] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250116 02:58:16.860567 20797 consensus_queue.cc:260] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.861580 20798 tablet_bootstrap.cc:492] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:16.865938 20862 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.867600 20868 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.858443 20799 consensus_queue.cc:260] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.877099 20868 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.879284 20799 ts_tablet_manager.cc:1428] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.026s user 0.007s sys 0.002s
I20250116 02:58:16.882952 20862 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.883206 20868 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:16.884697 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:16.883477 20798 tablet_bootstrap.cc:654] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.885409 20680 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:16.884244 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:16.886291 20682 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.886456 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:16.885646 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:16.887941 20607 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:16.889218 20723 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:16.890107 20868 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:16.890534 20868 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:16.890923 20868 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.891954 20799 tablet_bootstrap.cc:492] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:16.892520 20797 ts_tablet_manager.cc:1428] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20250116 02:58:16.893776 20682 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
W20250116 02:58:16.894708 20572 leader_election.cc:343] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.896303 20571 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:16.897691 20797 tablet_bootstrap.cc:492] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:16.902213 20863 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:16.902789 20863 raft_consensus.cc:695] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:16.903690 20863 consensus_queue.cc:237] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.914670 20798 tablet_bootstrap.cc:492] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:16.915243 20798 ts_tablet_manager.cc:1397] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.001s
I20250116 02:58:16.915891 20862 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250116 02:58:16.919149 20798 raft_consensus.cc:357] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.920166 20798 raft_consensus.cc:383] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:16.920451 20798 raft_consensus.cc:738] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:16.920722 20869 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.921110 20798 consensus_queue.cc:260] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.921247 20869 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.923061 20799 tablet_bootstrap.cc:654] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.923484 20869 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:16.924671 20868 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.925452 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:16.926174 20607 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:16.925868 20506 catalog_manager.cc:5526] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:16.927675 20868 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:16.927821 20723 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:16.928634 20868 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:16.929057 20868 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:16.929447 20868 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.929488 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:16.930199 20607 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.937449 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:16.938217 20681 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.943058 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:16.943954 20680 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:16.944234 20797 tablet_bootstrap.cc:654] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:16.946560 20863 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.947285 20863 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.951143 20681 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:16.937284 20607 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:16.953207 20939 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:16.937433 20868 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.954280 20723 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:16.953730 20939 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.956362 20939 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.956465 20868 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:16.957280 20868 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:16.957134 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:16.957911 20868 raft_consensus.cc:695] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:16.957970 20606 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:16.958588 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:16.959046 20868 consensus_queue.cc:237] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:16.961194 20649 leader_election.cc:343] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:16.966933 20939 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:16.968163 20606 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:16.969256 20723 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:16.971499 20863 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:16.971697 21008 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:16.972343 21008 raft_consensus.cc:695] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:16.973366 21008 consensus_queue.cc:237] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:16.943454 20798 ts_tablet_manager.cc:1428] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20250116 02:58:16.983737 20798 tablet_bootstrap.cc:492] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:16.993947 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.001045 20862 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.001617 20862 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.003945 20862 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:58:17.005635 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 67620.8us
I20250116 02:58:17.006721 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.011097 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:17.011799 20681 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.012580 20505 catalog_manager.cc:5526] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:17.014006 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:17.014730 20753 raft_consensus.cc:2463] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:17.018906 20681 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:17.019635 20799 tablet_bootstrap.cc:492] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:17.020165 20799 ts_tablet_manager.cc:1397] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.128s user 0.015s sys 0.002s
I20250116 02:58:17.022204 20797 tablet_bootstrap.cc:492] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:17.022835 20797 ts_tablet_manager.cc:1397] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.125s user 0.016s sys 0.018s
I20250116 02:58:17.023715 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:17.025620 20797 raft_consensus.cc:357] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.026405 20797 raft_consensus.cc:383] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.026772 20797 raft_consensus.cc:738] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.026472 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:17.027524 20797 consensus_queue.cc:260] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.029980 20797 ts_tablet_manager.cc:1428] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250116 02:58:17.030917 20797 tablet_bootstrap.cc:492] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:17.011112 20798 tablet_bootstrap.cc:654] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.042085 20572 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:17.043022 20799 raft_consensus.cc:357] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:17.043668 20572 consensus_peers.cc:487] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Couldn't send request to peer 17446c8dacf04461acb843547665250f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:17.044250 20799 raft_consensus.cc:383] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.044565 20799 raft_consensus.cc:738] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.044667 20798 tablet_bootstrap.cc:492] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:17.045256 20798 ts_tablet_manager.cc:1397] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.062s user 0.014s sys 0.002s
I20250116 02:58:17.045300 20799 consensus_queue.cc:260] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.047024 20647 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:17.047348 20799 ts_tablet_manager.cc:1428] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20250116 02:58:17.048389 20799 tablet_bootstrap.cc:492] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:17.048930 20939 raft_consensus.cc:2798] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:17.049384 20939 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:17.049731 20939 raft_consensus.cc:3054] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.052424 20797 tablet_bootstrap.cc:654] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.048324 20798 raft_consensus.cc:357] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.053296 20798 raft_consensus.cc:383] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.053632 20798 raft_consensus.cc:738] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.054469 20798 consensus_queue.cc:260] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.055992 20799 tablet_bootstrap.cc:654] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.057078 20798 ts_tablet_manager.cc:1428] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250116 02:58:17.044417 20572 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:17.045243 21004 raft_consensus.cc:2743] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.059413 20862 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.058302 20798 tablet_bootstrap.cc:492] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:17.063472 20506 catalog_manager.cc:5526] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:17.069444 20798 tablet_bootstrap.cc:654] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.073326 20797 tablet_bootstrap.cc:492] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:17.067854 20939 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.073855 20797 ts_tablet_manager.cc:1397] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.000s
I20250116 02:58:17.076431 20797 raft_consensus.cc:357] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.077018 20797 raft_consensus.cc:383] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.077255 20797 raft_consensus.cc:738] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.077833 20797 consensus_queue.cc:260] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.079650 20797 ts_tablet_manager.cc:1428] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250116 02:58:17.080540 20797 tablet_bootstrap.cc:492] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:17.086355 20799 tablet_bootstrap.cc:492] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:17.086876 20799 ts_tablet_manager.cc:1397] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.039s user 0.023s sys 0.003s
I20250116 02:58:17.089656 20799 raft_consensus.cc:357] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.090333 20797 tablet_bootstrap.cc:654] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.090397 20799 raft_consensus.cc:383] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.090951 20799 raft_consensus.cc:738] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.091725 20799 consensus_queue.cc:260] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.094641 20799 ts_tablet_manager.cc:1428] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20250116 02:58:17.095707 20799 tablet_bootstrap.cc:492] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:17.095974 20798 tablet_bootstrap.cc:492] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:17.096561 20798 ts_tablet_manager.cc:1397] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.000s
I20250116 02:58:17.099676 20798 raft_consensus.cc:357] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.100112 20810 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.099901 20882 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:17.100728 20798 raft_consensus.cc:383] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.100843 20810 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.101361 20798 raft_consensus.cc:738] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.102340 20798 consensus_queue.cc:260] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.103168 20810 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.105216 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:17.107004 20649 leader_election.cc:343] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.108644 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:17.110505 20799 tablet_bootstrap.cc:654] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
W20250116 02:58:17.111044 20647 leader_election.cc:343] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.111614 20647 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:17.112561 20810 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.118153 20798 ts_tablet_manager.cc:1428] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250116 02:58:17.119586 20606 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.129994 20797 tablet_bootstrap.cc:492] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:17.132436 20798 tablet_bootstrap.cc:492] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:17.137333 20810 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:17.138214 20797 ts_tablet_manager.cc:1397] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.058s user 0.020s sys 0.025s
I20250116 02:58:17.150687 20799 tablet_bootstrap.cc:492] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:17.151266 20799 ts_tablet_manager.cc:1397] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.007s
I20250116 02:58:17.153270 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:17.154067 20607 raft_consensus.cc:3054] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.154934 20797 raft_consensus.cc:357] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.155774 20797 raft_consensus.cc:383] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.156051 20797 raft_consensus.cc:738] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.156755 20797 consensus_queue.cc:260] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.159250 20797 ts_tablet_manager.cc:1428] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20250116 02:58:17.160288 20797 tablet_bootstrap.cc:492] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:17.163471 20607 raft_consensus.cc:2463] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:17.165100 20649 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:17.165729 20797 tablet_bootstrap.cc:654] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.166095 20810 raft_consensus.cc:2798] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:17.166590 20810 raft_consensus.cc:695] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:17.167532 20810 consensus_queue.cc:237] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.167563 20799 raft_consensus.cc:357] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.169414 20798 tablet_bootstrap.cc:654] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.169370 20799 raft_consensus.cc:383] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.171718 20799 raft_consensus.cc:738] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.172508 20799 consensus_queue.cc:260] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.175089 20799 ts_tablet_manager.cc:1428] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20250116 02:58:17.175292 20939 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.176082 20799 tablet_bootstrap.cc:492] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:17.176995 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:17.177682 20753 raft_consensus.cc:3054] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.183188 20799 tablet_bootstrap.cc:654] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.191282 20607 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.217792 20869 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.218472 20869 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.220677 20869 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:17.221603 20505 catalog_manager.cc:5526] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:17.222558 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:17.225088 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.225812 20682 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
W20250116 02:58:17.226466 20723 leader_election.cc:343] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.232558 20753 raft_consensus.cc:2463] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:17.240382 20797 tablet_bootstrap.cc:492] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:17.240554 20720 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:17.241031 20797 ts_tablet_manager.cc:1397] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.081s user 0.023s sys 0.012s
I20250116 02:58:17.247167 20939 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.253533 20869 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:17.254074 20869 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:17.253700 20939 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.254457 20869 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.257221 21010 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.258227 21010 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.260236 21010 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:17.261111 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:17.263466 20798 tablet_bootstrap.cc:492] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:17.263895 20798 ts_tablet_manager.cc:1397] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.132s user 0.026s sys 0.006s
I20250116 02:58:17.264477 20869 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.266134 20799 tablet_bootstrap.cc:492] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:17.266662 20799 ts_tablet_manager.cc:1397] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.025s
I20250116 02:58:17.266639 20797 raft_consensus.cc:357] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.267477 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:17.270267 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:17.280546 20797 raft_consensus.cc:383] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.272593 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.280965 20797 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
W20250116 02:58:17.281863 20649 leader_election.cc:343] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.281708 20797 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.283344 20720 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:17.284080 20797 ts_tablet_manager.cc:1428] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20250116 02:58:17.284587 21010 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.271103 20799 raft_consensus.cc:357] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.285063 20799 raft_consensus.cc:383] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.285419 20797 tablet_bootstrap.cc:492] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:17.285367 20799 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
W20250116 02:58:17.285534 20723 leader_election.cc:343] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.285302 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:17.287091 20799 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:17.288791 20647 leader_election.cc:343] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.289252 20647 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:17.280063 20939 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.290046 20969 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.289649 20799 ts_tablet_manager.cc:1428] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20250116 02:58:17.295528 20869 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:17.279467 20798 raft_consensus.cc:357] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.298521 20798 raft_consensus.cc:383] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.298872 20798 raft_consensus.cc:738] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.298939 20799 tablet_bootstrap.cc:492] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:17.300717 20798 consensus_queue.cc:260] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.303292 20798 ts_tablet_manager.cc:1428] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20250116 02:58:17.307098 21004 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:17.311038 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:17.311749 20682 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.312062 20799 tablet_bootstrap.cc:654] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.317807 20798 tablet_bootstrap.cc:492] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:17.323211 20797 tablet_bootstrap.cc:654] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.324468 20682 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:17.324971 20798 tablet_bootstrap.cc:654] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.325843 20720 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:17.326337 20799 tablet_bootstrap.cc:492] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:17.326952 20799 ts_tablet_manager.cc:1397] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.006s
I20250116 02:58:17.326952 20869 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:17.327818 20869 raft_consensus.cc:695] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:17.328747 20869 consensus_queue.cc:237] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.330055 20799 raft_consensus.cc:357] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.330834 20799 raft_consensus.cc:383] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.331117 20799 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.331770 20799 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.334197 20799 ts_tablet_manager.cc:1428] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250116 02:58:17.334967 20797 tablet_bootstrap.cc:492] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:17.335256 20799 tablet_bootstrap.cc:492] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:17.335645 20797 ts_tablet_manager.cc:1397] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.001s
I20250116 02:58:17.338037 20797 raft_consensus.cc:357] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.339097 20797 raft_consensus.cc:383] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.339395 20797 raft_consensus.cc:738] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.339993 20797 consensus_queue.cc:260] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.342445 20797 ts_tablet_manager.cc:1428] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250116 02:58:17.342725 20799 tablet_bootstrap.cc:654] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.348182 20753 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.358973 21004 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.359697 21004 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.362004 20797 tablet_bootstrap.cc:492] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:17.361923 20799 tablet_bootstrap.cc:492] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:17.364070 20799 ts_tablet_manager.cc:1397] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.005s
I20250116 02:58:17.364076 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.363468 21004 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.364692 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:17.365960 20755 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:17.365963 20681 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:17.367425 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:17.368304 21004 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:17.368781 21004 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:17.368400 20799 raft_consensus.cc:357] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.369208 21004 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.369271 20799 raft_consensus.cc:383] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.369659 20799 raft_consensus.cc:738] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.370353 20799 consensus_queue.cc:260] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.372738 20799 ts_tablet_manager.cc:1428] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250116 02:58:17.373922 20799 tablet_bootstrap.cc:492] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:17.374243 20887 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:17.368424 20505 catalog_manager.cc:5526] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:17.377879 21004 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.380389 20799 tablet_bootstrap.cc:654] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.382067 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:17.382843 20681 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.393532 20887 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.394204 20887 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.396549 20862 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.397130 20862 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.398151 20887 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.399281 20862 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.405066 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.406816 20683 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:17.402045 20681 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:17.408390 21004 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.404990 21008 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250116 02:58:17.411340 20798 tablet_bootstrap.cc:492] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:17.411897 20798 ts_tablet_manager.cc:1397] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.094s user 0.016s sys 0.014s
I20250116 02:58:17.403087 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.406682 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:17.411368 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:17.416639 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:17.417510 20799 tablet_bootstrap.cc:492] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:17.417964 20799 ts_tablet_manager.cc:1397] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.002s
I20250116 02:58:17.415984 20798 raft_consensus.cc:357] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.419417 20798 raft_consensus.cc:383] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.419624 20798 raft_consensus.cc:738] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.420118 20798 consensus_queue.cc:260] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.421320 20798 ts_tablet_manager.cc:1428] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
W20250116 02:58:17.423483 20572 leader_election.cc:343] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.423765 20798 tablet_bootstrap.cc:492] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:17.417326 20752 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.429849 20797 tablet_bootstrap.cc:654] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.427587 20571 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:17.431721 21004 raft_consensus.cc:2798] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:17.432116 21004 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:17.432468 21004 raft_consensus.cc:3054] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:17.434144 20571 leader_election.cc:343] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.435148 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:17.440490 21008 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.440658 20862 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:17.440856 20798 tablet_bootstrap.cc:654] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.441114 21008 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.442965 20862 raft_consensus.cc:695] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:58:17.443593 20572 leader_election.cc:343] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.445206 20572 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:17.445377 20862 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.446108 20887 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.446576 21008 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:17.450812 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:17.451514 20607 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:17.452343 20799 raft_consensus.cc:357] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.452927 20723 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:17.454774 20799 raft_consensus.cc:383] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.455178 20799 raft_consensus.cc:738] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.453536 20752 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:17.455828 20799 consensus_queue.cc:260] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.455256 20869 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:17.456882 20869 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:17.457247 20869 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.457557 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.458459 20680 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:17.463155 21004 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.463894 21008 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:17.469383 21010 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:17.470942 20799 ts_tablet_manager.cc:1428] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.053s user 0.008s sys 0.001s
I20250116 02:58:17.472079 20752 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.474573 20799 tablet_bootstrap.cc:492] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:17.480934 20869 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.484593 20869 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:17.485005 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:17.485730 20608 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.490731 20863 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.491318 20863 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.492357 20798 tablet_bootstrap.cc:492] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:17.492858 20798 ts_tablet_manager.cc:1397] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.069s user 0.007s sys 0.005s
I20250116 02:58:17.490795 20799 tablet_bootstrap.cc:654] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.493474 20863 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.494681 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:17.499966 20755 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:17.501250 20572 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:17.502177 20863 raft_consensus.cc:2798] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:17.502543 20863 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:17.502882 20863 raft_consensus.cc:3054] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.503090 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:17.503731 20680 raft_consensus.cc:3054] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.503772 20798 raft_consensus.cc:357] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.504766 20798 raft_consensus.cc:383] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.505138 20798 raft_consensus.cc:738] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.505997 20798 consensus_queue.cc:260] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.509532 20798 ts_tablet_manager.cc:1428] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20250116 02:58:17.510824 20798 tablet_bootstrap.cc:492] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:17.511483 20680 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:17.512223 20799 tablet_bootstrap.cc:492] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:17.512717 20799 ts_tablet_manager.cc:1397] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.003s
I20250116 02:58:17.511077 20863 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.513008 20571 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:17.513942 20862 raft_consensus.cc:2798] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:17.514443 20862 raft_consensus.cc:695] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:17.502828 20506 catalog_manager.cc:5526] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:17.515381 20862 consensus_queue.cc:237] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.515803 20799 raft_consensus.cc:357] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.516631 20799 raft_consensus.cc:383] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.516935 20799 raft_consensus.cc:738] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.517887 20799 consensus_queue.cc:260] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.520448 20797 tablet_bootstrap.cc:492] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:17.521016 20797 ts_tablet_manager.cc:1397] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.177s user 0.012s sys 0.000s
I20250116 02:58:17.498065 20608 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:17.525780 20723 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:17.526881 20869 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:17.524464 21004 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.527457 20869 raft_consensus.cc:695] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:17.527330 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:17.528451 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:17.529201 20681 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.528927 20869 consensus_queue.cc:237] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:17.529728 20572 leader_election.cc:343] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.532459 20799 ts_tablet_manager.cc:1428] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.019s user 0.004s sys 0.004s
I20250116 02:58:17.533924 20799 tablet_bootstrap.cc:492] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:17.536268 20863 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.537810 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:17.538997 20752 raft_consensus.cc:3054] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.540617 20798 tablet_bootstrap.cc:654] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.543013 20797 raft_consensus.cc:357] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.543815 20797 raft_consensus.cc:383] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.544118 20797 raft_consensus.cc:738] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.544672 20797 consensus_queue.cc:260] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.552434 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.556420 20682 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:17.558657 21008 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.559779 21008 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.563025 21008 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:17.564117 20505 catalog_manager.cc:5526] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:17.564605 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:17.568601 20752 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:17.569880 20797 ts_tablet_manager.cc:1428] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.049s user 0.005s sys 0.002s
I20250116 02:58:17.576009 20797 tablet_bootstrap.cc:492] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:17.582358 20799 tablet_bootstrap.cc:654] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.586059 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:17.586853 20682 raft_consensus.cc:3054] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.587834 20572 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:17.596081 20887 raft_consensus.cc:2798] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:17.583683 21010 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20250116 02:58:17.601481 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.598414 20887 raft_consensus.cc:695] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:17.614566 20720 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:17.619632 20887 consensus_queue.cc:237] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.633365 20682 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
W20250116 02:58:17.634740 21044 log.cc:926] Time spent T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20250116 02:58:17.639834 20797 tablet_bootstrap.cc:654] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.639564 20681 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:17.620810 20869 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.642727 21004 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.654136 21004 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.688956 21004 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.690665 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:17.699472 20572 leader_election.cc:343] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.729271 20798 tablet_bootstrap.cc:492] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:17.729871 20798 ts_tablet_manager.cc:1397] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.219s user 0.012s sys 0.005s
I20250116 02:58:17.703733 21047 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.747627 21047 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.750123 21047 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:17.745384 20797 tablet_bootstrap.cc:492] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:17.734683 20882 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:17.725556 20939 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.753372 20798 raft_consensus.cc:357] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.759009 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:17.763420 20797 ts_tablet_manager.cc:1397] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.188s user 0.019s sys 0.004s
I20250116 02:58:17.767042 20798 raft_consensus.cc:383] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.770263 20798 raft_consensus.cc:738] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.770996 20798 consensus_queue.cc:260] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.775167 20798 ts_tablet_manager.cc:1428] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.045s user 0.008s sys 0.001s
I20250116 02:58:17.775633 20797 raft_consensus.cc:357] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.776443 20797 raft_consensus.cc:383] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:17.776705 20797 raft_consensus.cc:738] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.777343 20797 consensus_queue.cc:260] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.778321 20798 tablet_bootstrap.cc:492] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:17.787110 20506 catalog_manager.cc:5526] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:17.788130 20504 catalog_manager.cc:5526] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:17.770067 20939 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:17.782195 20723 leader_election.cc:343] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.792431 20797 ts_tablet_manager.cc:1428] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250116 02:58:17.794581 20797 tablet_bootstrap.cc:492] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:17.795814 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.796753 20939 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.805488 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:17.806329 20605 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:17.808213 20649 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:17.813009 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:17.849113 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:17.851079 20799 tablet_bootstrap.cc:492] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:17.851715 20799 ts_tablet_manager.cc:1397] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.318s user 0.019s sys 0.008s
I20250116 02:58:17.856729 21010 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.861114 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.861515 21010 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.857154 21053 raft_consensus.cc:2798] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250116 02:58:17.854921 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 46823us
I20250116 02:58:17.870198 20798 tablet_bootstrap.cc:654] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
W20250116 02:58:17.870550 20647 leader_election.cc:343] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.870589 21051 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:17.876379 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:17.866087 21053 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:17.879336 21053 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.880683 20799 raft_consensus.cc:357] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:17.883772 20723 leader_election.cc:343] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.884483 21010 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:17.882097 20799 raft_consensus.cc:383] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250116 02:58:17.884456 20720 leader_election.cc:343] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.885455 20799 raft_consensus.cc:738] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:17.885565 20720 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:17.890051 21010 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.885095 20755 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.892592 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.889633 20799 consensus_queue.cc:260] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.893357 20682 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:17.898108 21053 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.906504 20680 raft_consensus.cc:2388] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:17.889523 20810 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.908809 20810 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.899381 20799 ts_tablet_manager.cc:1428] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.047s user 0.008s sys 0.000s
I20250116 02:58:17.911281 21053 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.905380 20797 tablet_bootstrap.cc:654] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:17.910485 20720 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:17.915184 20868 raft_consensus.cc:2798] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:17.915601 20868 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:17.915974 20868 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.916795 20799 tablet_bootstrap.cc:492] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:17.919770 20571 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:17.921140 20887 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.921581 20810 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.923574 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:17.925261 20862 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250116 02:58:17.928601 20647 leader_election.cc:343] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.932520 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:17.942800 20647 leader_election.cc:343] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.943240 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:17.944109 20607 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:17.946936 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:17.947775 21010 consensus_queue.cc:1035] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250116 02:58:17.951602 20887 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.956984 20798 tablet_bootstrap.cc:492] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:17.957549 20798 ts_tablet_manager.cc:1397] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.180s user 0.019s sys 0.005s
W20250116 02:58:17.961633 20723 consensus_peers.cc:487] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f -> Peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Couldn't send request to peer f65e8d7f76144fab9725ca32a6328c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:17.959534 20862 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:17.965904 20862 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.968317 20862 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:17.969470 20649 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:17.970978 20810 raft_consensus.cc:2798] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:17.971379 20810 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:17.971714 20810 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:17.954660 20868 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:17.976073 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:17.976678 20868 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:17.976958 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:17.978431 20572 leader_election.cc:343] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.978533 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:58:17.983978 20571 leader_election.cc:343] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.988260 20571 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:17.989558 20887 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:17.984496 20606 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
W20250116 02:58:17.998136 20723 leader_election.cc:343] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:17.999490 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:18.000150 20682 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.003026 20649 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:18.012280 20797 tablet_bootstrap.cc:492] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:18.012971 20797 ts_tablet_manager.cc:1397] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.219s user 0.008s sys 0.012s
I20250116 02:58:18.015002 20882 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.015648 20882 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.014161 21053 raft_consensus.cc:2798] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:18.022794 20882 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.032177 20799 tablet_bootstrap.cc:654] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.031875 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.038798 21053 raft_consensus.cc:695] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:18.041361 21053 consensus_queue.cc:237] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.044975 20887 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250116 02:58:18.046655 20649 leader_election.cc:343] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:18.050591 20810 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.053018 20810 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.043222 20682 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:18.056464 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:18.057866 20607 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.053804 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:18.069330 20647 leader_election.cc:343] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:18.069825 20647 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:18.069989 20720 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:18.056586 20798 raft_consensus.cc:357] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.056550 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:18.071296 20810 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:18.072569 20798 raft_consensus.cc:383] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.074227 20798 raft_consensus.cc:738] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.075147 20798 consensus_queue.cc:260] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:18.075845 20647 leader_election.cc:343] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:18.077183 20607 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:18.076310 21010 raft_consensus.cc:2798] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:18.083151 20649 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:18.084307 21051 raft_consensus.cc:2798] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:18.084944 21051 raft_consensus.cc:695] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:18.085911 21051 consensus_queue.cc:237] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.081765 21010 raft_consensus.cc:695] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:18.088276 21010 consensus_queue.cc:237] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.091959 20797 raft_consensus.cc:357] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.093197 20797 raft_consensus.cc:383] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.096814 20797 raft_consensus.cc:738] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.102209 21047 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250116 02:58:18.105643 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 72611.5us
I20250116 02:58:18.123838 20868 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:18.133287 21064 consensus_queue.cc:1035] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:18.137208 20797 consensus_queue.cc:260] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.141471 20753 raft_consensus.cc:3054] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.094777 20798 ts_tablet_manager.cc:1428] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.137s user 0.008s sys 0.000s
I20250116 02:58:18.148325 20798 tablet_bootstrap.cc:492] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:18.141685 20797 ts_tablet_manager.cc:1428] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.128s user 0.007s sys 0.002s
I20250116 02:58:18.152761 20797 tablet_bootstrap.cc:492] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:18.155658 21064 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:18.157296 20504 catalog_manager.cc:5526] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:18.160580 20862 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:18.161144 20799 tablet_bootstrap.cc:492] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:18.161701 20799 ts_tablet_manager.cc:1397] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.245s user 0.013s sys 0.004s
I20250116 02:58:18.164867 20799 raft_consensus.cc:357] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.165740 20799 raft_consensus.cc:383] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.166047 20799 raft_consensus.cc:738] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.171478 20797 tablet_bootstrap.cc:654] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.172355 20799 consensus_queue.cc:260] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.175614 20799 ts_tablet_manager.cc:1428] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.014s user 0.007s sys 0.002s
I20250116 02:58:18.176683 20799 tablet_bootstrap.cc:492] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:18.181751 20798 tablet_bootstrap.cc:654] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.185810 20799 tablet_bootstrap.cc:654] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.187067 21063 consensus_queue.cc:1035] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:18.187963 20506 catalog_manager.cc:5526] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:18.190434 21047 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.191064 21047 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.193219 21047 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:18.204192 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.207410 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:18.213665 20720 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:18.229174 21008 raft_consensus.cc:2743] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:58:18.241335 20799 tablet_bootstrap.cc:492] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:18.241989 20799 ts_tablet_manager.cc:1397] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.065s user 0.011s sys 0.000s
I20250116 02:58:18.244889 20799 raft_consensus.cc:357] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.245710 20799 raft_consensus.cc:383] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.246003 20799 raft_consensus.cc:738] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.246132 20798 tablet_bootstrap.cc:492] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:18.246663 20798 ts_tablet_manager.cc:1397] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.099s user 0.011s sys 0.015s
I20250116 02:58:18.246789 20799 consensus_queue.cc:260] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.250262 20799 ts_tablet_manager.cc:1428] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250116 02:58:18.250815 20797 tablet_bootstrap.cc:492] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:18.251323 20797 ts_tablet_manager.cc:1397] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.099s user 0.027s sys 0.016s
I20250116 02:58:18.251400 20799 tablet_bootstrap.cc:492] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:18.254091 20797 raft_consensus.cc:357] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.254933 20797 raft_consensus.cc:383] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.255519 20797 raft_consensus.cc:738] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.256213 20797 consensus_queue.cc:260] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.256583 20798 raft_consensus.cc:357] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.257531 20798 raft_consensus.cc:383] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.257807 20798 raft_consensus.cc:738] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.258392 20798 consensus_queue.cc:260] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.260293 21051 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.262331 21051 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.264096 20797 ts_tablet_manager.cc:1428] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250116 02:58:18.265136 20797 tablet_bootstrap.cc:492] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:18.268188 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.262653 20799 tablet_bootstrap.cc:654] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.260531 20798 ts_tablet_manager.cc:1428] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250116 02:58:18.276737 21051 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.265947 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:18.280095 20752 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:18.281473 20647 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:18.275383 20797 tablet_bootstrap.cc:654] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.282914 21051 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:18.283339 21051 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:18.284127 21051 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.284804 20606 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:18.286495 20798 tablet_bootstrap.cc:492] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:18.288298 20799 tablet_bootstrap.cc:492] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:18.289012 20799 ts_tablet_manager.cc:1397] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.002s
I20250116 02:58:18.292883 20798 tablet_bootstrap.cc:654] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.275373 20506 catalog_manager.cc:5526] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:18.293102 20799 raft_consensus.cc:357] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.298748 20799 raft_consensus.cc:383] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.299108 20799 raft_consensus.cc:738] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.294404 21051 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.302032 20797 tablet_bootstrap.cc:492] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:18.302572 20797 ts_tablet_manager.cc:1397] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.007s
I20250116 02:58:18.304025 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:18.304716 20606 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.305596 20797 raft_consensus.cc:357] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.306435 20797 raft_consensus.cc:383] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.306771 20797 raft_consensus.cc:738] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.307513 20797 consensus_queue.cc:260] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.310017 20797 ts_tablet_manager.cc:1428] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250116 02:58:18.312889 20606 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:18.314924 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:18.315613 20649 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:18.313822 21051 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.315631 20753 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.299793 20799 consensus_queue.cc:260] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.316802 21051 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:18.317862 21051 raft_consensus.cc:695] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:18.319123 20799 ts_tablet_manager.cc:1428] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.030s user 0.004s sys 0.003s
I20250116 02:58:18.318892 21051 consensus_queue.cc:237] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.320199 20799 tablet_bootstrap.cc:492] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:18.325489 20753 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:18.327313 20797 tablet_bootstrap.cc:492] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:18.328874 20799 tablet_bootstrap.cc:654] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.357597 20797 tablet_bootstrap.cc:654] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.362212 20798 tablet_bootstrap.cc:492] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:18.363554 20798 ts_tablet_manager.cc:1397] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.077s user 0.024s sys 0.008s
I20250116 02:58:18.364692 20799 tablet_bootstrap.cc:492] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:18.363988 20506 catalog_manager.cc:5526] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:18.365214 20799 ts_tablet_manager.cc:1397] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.004s
I20250116 02:58:18.367266 20798 raft_consensus.cc:357] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.368207 20798 raft_consensus.cc:383] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.368541 20798 raft_consensus.cc:738] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.369287 20798 consensus_queue.cc:260] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.371865 20798 ts_tablet_manager.cc:1428] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250116 02:58:18.372932 20798 tablet_bootstrap.cc:492] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:18.374464 20606 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.375495 20797 tablet_bootstrap.cc:492] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:18.376011 20797 ts_tablet_manager.cc:1397] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.008s
I20250116 02:58:18.378474 20799 raft_consensus.cc:357] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.379565 20799 raft_consensus.cc:383] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.379855 20799 raft_consensus.cc:738] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.380647 20798 tablet_bootstrap.cc:654] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.380544 20799 consensus_queue.cc:260] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.384303 20797 raft_consensus.cc:357] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.385356 20797 raft_consensus.cc:383] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.385644 20797 raft_consensus.cc:738] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.386324 20797 consensus_queue.cc:260] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.389312 20797 ts_tablet_manager.cc:1428] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250116 02:58:18.389833 20799 ts_tablet_manager.cc:1428] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20250116 02:58:18.390311 20797 tablet_bootstrap.cc:492] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:18.390946 20799 tablet_bootstrap.cc:492] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:18.400043 20797 tablet_bootstrap.cc:654] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.400914 21010 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.401135 20799 tablet_bootstrap.cc:654] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.411842 20797 tablet_bootstrap.cc:492] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:18.412405 20797 ts_tablet_manager.cc:1397] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.003s
I20250116 02:58:18.413133 20798 tablet_bootstrap.cc:492] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:18.413790 20798 ts_tablet_manager.cc:1397] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.000s
I20250116 02:58:18.415211 20862 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.401518 21010 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.415786 20862 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.417531 20799 tablet_bootstrap.cc:492] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:18.418066 20799 ts_tablet_manager.cc:1397] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.006s
I20250116 02:58:18.421856 20797 raft_consensus.cc:357] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.422013 20798 raft_consensus.cc:357] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.422854 20797 raft_consensus.cc:383] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.422940 20798 raft_consensus.cc:383] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.423223 20797 raft_consensus.cc:738] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.423902 20797 consensus_queue.cc:260] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.424171 20799 raft_consensus.cc:357] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.424909 20799 raft_consensus.cc:383] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.425202 20799 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.417521 21010 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:18.426440 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.426589 20797 ts_tablet_manager.cc:1428] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250116 02:58:18.422040 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:18.427289 20607 raft_consensus.cc:2388] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:18.425895 20798 raft_consensus.cc:738] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.428524 20798 consensus_queue.cc:260] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.427750 20797 tablet_bootstrap.cc:492] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:18.430898 20798 ts_tablet_manager.cc:1428] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.017s user 0.004s sys 0.003s
I20250116 02:58:18.425833 20799 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.434396 20798 tablet_bootstrap.cc:492] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:18.441571 20798 tablet_bootstrap.cc:654] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.432725 20720 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:18.446509 20799 ts_tablet_manager.cc:1428] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.028s user 0.003s sys 0.005s
I20250116 02:58:18.450465 20869 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.450955 20799 tablet_bootstrap.cc:492] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:18.457971 20799 tablet_bootstrap.cc:654] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.459142 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:18.447348 20862 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.460418 20797 tablet_bootstrap.cc:654] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.457873 20869 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:18.462280 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:18.462280 20682 raft_consensus.cc:2388] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:58:18.465687 21051 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.467661 21051 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.471500 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:18.472330 20755 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:18.473017 20572 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:18.473951 20862 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.476132 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.477011 20607 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:18.479820 20647 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:18.481498 20939 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:18.481894 20939 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:18.482232 20939 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.488206 20862 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:18.491603 21047 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.492245 20798 tablet_bootstrap.cc:492] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:18.492460 21051 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.492213 21047 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.494143 21010 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.494691 20797 tablet_bootstrap.cc:492] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:18.494699 21010 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.496254 20797 ts_tablet_manager.cc:1397] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.068s user 0.011s sys 0.004s
I20250116 02:58:18.495802 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:18.497342 20798 ts_tablet_manager.cc:1397] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.063s user 0.029s sys 0.013s
I20250116 02:58:18.494488 21047 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:18.499604 21010 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:18.502193 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:18.504827 20798 raft_consensus.cc:357] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.505322 20799 tablet_bootstrap.cc:492] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:18.505646 20798 raft_consensus.cc:383] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.505971 20798 raft_consensus.cc:738] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.506059 20799 ts_tablet_manager.cc:1397] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.000s
W20250116 02:58:18.506218 20720 leader_election.cc:343] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:18.506601 20798 consensus_queue.cc:260] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.507668 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.508431 20607 raft_consensus.cc:2388] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:18.508656 20798 ts_tablet_manager.cc:1428] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250116 02:58:18.512046 20797 raft_consensus.cc:357] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.512861 20797 raft_consensus.cc:383] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.513150 20797 raft_consensus.cc:738] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.513834 20797 consensus_queue.cc:260] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.514658 20798 tablet_bootstrap.cc:492] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:18.517858 20799 raft_consensus.cc:357] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.518692 20799 raft_consensus.cc:383] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:18.519001 20799 raft_consensus.cc:738] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:18.519490 20863 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.522107 21010 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:18.522982 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.522884 20863 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.523720 20607 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:18.529487 20797 ts_tablet_manager.cc:1428] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.033s user 0.002s sys 0.004s
I20250116 02:58:18.522197 20723 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:18.534368 20939 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.543069 20798 tablet_bootstrap.cc:654] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.544370 20799 consensus_queue.cc:260] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.544931 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:18.545585 20607 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.546952 20799 ts_tablet_manager.cc:1428] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.041s user 0.008s sys 0.001s
I20250116 02:58:18.548205 20797 tablet_bootstrap.cc:492] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:18.535625 20863 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.530936 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:18.537271 21047 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:18.538552 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:18.539789 20723 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:18.540946 21008 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.550415 20799 tablet_bootstrap.cc:492] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:18.555589 20607 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:18.556866 20939 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.552376 20682 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:18.557345 20969 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.558890 20969 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:18.561722 20572 leader_election.cc:343] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:18.563730 20649 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:18.566061 21051 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:18.566649 21051 raft_consensus.cc:695] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:18.565204 20571 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:18.567557 21051 consensus_queue.cc:237] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.571735 21047 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:18.572175 21047 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:18.572289 20862 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:18.575006 20862 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:18.575413 20862 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.570345 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.578482 20607 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:18.579937 20649 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:18.582398 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:18.583335 20753 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.578325 21047 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.585937 20939 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.585727 21053 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:18.596541 21053 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:18.597992 21053 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.602453 20969 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.604003 20969 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.601071 21010 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:18.604496 20969 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.605180 21010 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.598779 20939 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:18.584256 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 44576.4us
I20250116 02:58:18.619086 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:18.620790 21010 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:18.621259 20939 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.622551 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:18.624109 20723 leader_election.cc:343] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:18.615406 20862 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.616886 20969 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.633188 20862 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:18.642693 20797 tablet_bootstrap.cc:654] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.819204 20799 tablet_bootstrap.cc:654] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:18.653434 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:18.836738 21047 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.855562 20752 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:58:18.656832 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:18.751989 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.763228 21053 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:18.900485 20753 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:18.718032 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:18.905333 20680 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:18.737743 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:18.911847 20607 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:18.913318 20798 tablet_bootstrap.cc:492] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:18.913878 20798 ts_tablet_manager.cc:1397] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.399s user 0.046s sys 0.048s
I20250116 02:58:18.926388 21047 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:18.925542 21008 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:18.930186 21053 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:58:18.991247 20572 leader_election.cc:343] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:18.993510 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 324953us
W20250116 02:58:19.011938 20720 leader_election.cc:343] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:19.015789 20720 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:19.015460 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:19.016533 20798 raft_consensus.cc:357] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.018155 20798 raft_consensus.cc:383] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.018323 20682 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:19.015494 20504 catalog_manager.cc:5526] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:19.028934 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:19.030277 20607 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:19.034925 20798 raft_consensus.cc:738] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.056419 21010 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:19.058113 20798 consensus_queue.cc:260] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.098440 20798 ts_tablet_manager.cc:1428] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.184s user 0.009s sys 0.002s
I20250116 02:58:19.114468 20607 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
W20250116 02:58:19.141311 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 129649us
W20250116 02:58:19.159303 20649 leader_election.cc:343] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:19.160411 20649 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:19.166939 20868 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:19.167933 20868 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.175271 20797 tablet_bootstrap.cc:492] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:19.178158 20868 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:19.178467 20798 tablet_bootstrap.cc:492] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:19.168042 20723 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:19.200873 20797 ts_tablet_manager.cc:1397] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.653s user 0.038s sys 0.005s
I20250116 02:58:19.202008 21008 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:19.202772 21008 raft_consensus.cc:695] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:19.204165 21008 consensus_queue.cc:237] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.220158 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:19.220935 20605 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:58:19.217078 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 49266.3us
I20250116 02:58:19.223982 20797 raft_consensus.cc:357] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.224900 20797 raft_consensus.cc:383] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.225142 20797 raft_consensus.cc:738] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.225728 20797 consensus_queue.cc:260] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.233261 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:19.234287 20607 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:19.241518 20680 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:19.233368 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:19.243211 20571 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:19.245007 21108 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:19.245643 21108 raft_consensus.cc:695] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:19.246871 21108 consensus_queue.cc:237] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.250792 21114 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:19.251281 21114 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:19.251688 21114 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:19.241386 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:19.254256 20755 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:19.268857 20797 ts_tablet_manager.cc:1428] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.068s user 0.006s sys 0.002s
I20250116 02:58:19.274168 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:19.278709 20797 tablet_bootstrap.cc:492] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:19.282670 20799 tablet_bootstrap.cc:492] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:19.279399 20723 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:19.289048 20868 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:19.289537 20868 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:19.289876 20868 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:19.291114 20798 tablet_bootstrap.cc:654] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:19.302881 20799 ts_tablet_manager.cc:1397] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.753s user 0.018s sys 0.076s
I20250116 02:58:19.313378 20799 raft_consensus.cc:357] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.314528 20799 raft_consensus.cc:383] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.314916 20799 raft_consensus.cc:738] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.315644 20799 consensus_queue.cc:260] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.319114 20799 ts_tablet_manager.cc:1428] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20250116 02:58:19.320606 20799 tablet_bootstrap.cc:492] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:19.309863 21114 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.328672 20868 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.335359 20868 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:19.337651 21103 consensus_queue.cc:1035] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:19.346390 20797 tablet_bootstrap.cc:654] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:19.370230 20939 consensus_queue.cc:1035] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:19.368067 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:19.374742 20606 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:19.373303 20939 consensus_queue.cc:1035] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250116 02:58:19.379431 21114 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:19.383193 20504 catalog_manager.cc:5526] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:19.390666 20799 tablet_bootstrap.cc:654] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
W20250116 02:58:19.392728 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:19.403684 20606 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:19.404294 21103 consensus_queue.cc:1035] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:19.405947 20723 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:19.406968 21010 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:19.407835 21010 raft_consensus.cc:695] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:19.407506 20503 catalog_manager.cc:5526] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:19.409021 21010 consensus_queue.cc:237] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.412173 21114 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:19.412849 21114 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.437835 21103 consensus_queue.cc:1035] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:19.443598 20797 tablet_bootstrap.cc:492] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:19.444254 20797 ts_tablet_manager.cc:1397] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.174s user 0.016s sys 0.003s
I20250116 02:58:19.448968 21104 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:19.449596 21104 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.453323 21104 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:19.455325 20798 tablet_bootstrap.cc:492] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:19.455978 20798 ts_tablet_manager.cc:1397] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.278s user 0.020s sys 0.002s
I20250116 02:58:19.462844 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:19.463891 20607 raft_consensus.cc:2388] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:58:19.463100 20647 leader_election.cc:343] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:19.464743 20647 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:19.465714 21103 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:19.466060 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:19.467090 20679 raft_consensus.cc:2388] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:19.469005 21096 consensus_queue.cc:1035] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:19.474681 21103 consensus_queue.cc:1035] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:19.478780 20647 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:19.480091 21103 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:58:19.486336 20503 catalog_manager.cc:5526] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:19.490516 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:19.493211 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:19.494133 20680 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:19.482512 20798 raft_consensus.cc:357] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.495524 20571 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:19.495934 20798 raft_consensus.cc:383] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.496270 20798 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.496430 21104 raft_consensus.cc:2798] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:19.497077 21104 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:19.496909 20798 consensus_queue.cc:260] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.497469 21104 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:19.499300 20798 ts_tablet_manager.cc:1428] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.043s user 0.008s sys 0.000s
I20250116 02:58:19.500294 20798 tablet_bootstrap.cc:492] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:19.504694 20799 tablet_bootstrap.cc:492] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:19.505299 20799 ts_tablet_manager.cc:1397] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.185s user 0.037s sys 0.054s
I20250116 02:58:19.508448 21110 consensus_queue.cc:1035] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:19.512876 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:19.513872 20679 raft_consensus.cc:2388] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:19.514544 21114 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:19.490518 20797 raft_consensus.cc:357] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.519472 20797 raft_consensus.cc:383] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.519790 20797 raft_consensus.cc:738] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.527448 20797 consensus_queue.cc:260] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.530047 20797 ts_tablet_manager.cc:1428] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.085s user 0.002s sys 0.005s
I20250116 02:58:19.530678 20799 raft_consensus.cc:357] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.536022 20799 raft_consensus.cc:383] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.536378 20799 raft_consensus.cc:738] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.537081 20799 consensus_queue.cc:260] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.539033 20797 tablet_bootstrap.cc:492] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
W20250116 02:58:19.539825 20649 leader_election.cc:343] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:19.543761 21104 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.548735 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:19.549520 20681 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:19.554916 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:19.557121 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:19.558660 20681 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:19.559976 20571 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:19.561007 20863 raft_consensus.cc:2798] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:19.561527 20863 raft_consensus.cc:695] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:19.562413 20863 consensus_queue.cc:237] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.570648 20799 ts_tablet_manager.cc:1428] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.065s user 0.007s sys 0.003s
I20250116 02:58:19.571866 20799 tablet_bootstrap.cc:492] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:19.578104 21104 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:19.586635 20799 tablet_bootstrap.cc:654] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:19.596746 20797 tablet_bootstrap.cc:654] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:19.636780 20647 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:58:19.638397 20647 leader_election.cc:343] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:19.638940 20647 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:19.648845 20798 tablet_bootstrap.cc:654] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:19.651048 21110 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:58:19.674918 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:19.676004 20756 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:19.680132 20799 tablet_bootstrap.cc:492] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:19.680742 20799 ts_tablet_manager.cc:1397] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.000s
I20250116 02:58:19.674700 20810 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:19.684823 20799 raft_consensus.cc:357] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.685635 20799 raft_consensus.cc:383] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.685931 20799 raft_consensus.cc:738] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.772645 20799 consensus_queue.cc:260] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:19.774446 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 125156us
W20250116 02:58:19.788863 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 139407us
I20250116 02:58:19.807986 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:19.808452 20799 ts_tablet_manager.cc:1428] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.127s user 0.007s sys 0.002s
I20250116 02:58:19.808845 21104 consensus_queue.cc:1035] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:19.809648 20799 tablet_bootstrap.cc:492] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:19.822778 21010 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:19.823472 21010 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.826804 21010 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:58:19.829540 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 99909.4us
I20250116 02:58:19.833043 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:19.819792 20756 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:19.841691 20799 tablet_bootstrap.cc:654] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:19.856249 20797 tablet_bootstrap.cc:492] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:19.856848 20797 ts_tablet_manager.cc:1397] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.318s user 0.010s sys 0.036s
I20250116 02:58:19.827571 21010 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20250116 02:58:19.872713 20797 raft_consensus.cc:357] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.873601 20797 raft_consensus.cc:383] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.873888 20797 raft_consensus.cc:738] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.875412 20797 consensus_queue.cc:260] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.875713 20503 catalog_manager.cc:5526] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:19.879880 20797 ts_tablet_manager.cc:1428] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.023s user 0.009s sys 0.001s
I20250116 02:58:19.881270 20797 tablet_bootstrap.cc:492] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:19.887405 20799 tablet_bootstrap.cc:492] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:19.888077 20799 ts_tablet_manager.cc:1397] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.006s
I20250116 02:58:19.887907 20756 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:19.894779 20799 raft_consensus.cc:357] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.896150 20799 raft_consensus.cc:383] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.896478 20799 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.895795 20798 tablet_bootstrap.cc:492] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:19.897469 20798 ts_tablet_manager.cc:1397] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.397s user 0.044s sys 0.019s
I20250116 02:58:19.898690 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:19.914798 20798 raft_consensus.cc:357] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.915725 20798 raft_consensus.cc:383] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:19.916023 20798 raft_consensus.cc:738] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:19.916428 21047 consensus_queue.cc:1035] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:19.916699 20798 consensus_queue.cc:260] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.899782 20799 consensus_queue.cc:260] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.926087 20798 ts_tablet_manager.cc:1428] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.028s user 0.007s sys 0.001s
I20250116 02:58:19.928589 20799 ts_tablet_manager.cc:1428] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.040s user 0.009s sys 0.000s
I20250116 02:58:19.940106 20798 tablet_bootstrap.cc:492] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:19.954114 20720 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:19.948393 20754 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:19.961541 21008 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:19.962123 21008 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.962447 21103 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:19.963232 21103 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:19.964321 21008 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:19.965261 21103 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:19.976226 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:19.980441 20608 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:19.987046 20723 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:19.987079 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:19.960906 20868 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20250116 02:58:19.998582 20797 tablet_bootstrap.cc:654] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.001991 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:20.002840 20756 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:20.008926 21047 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:20.018874 21010 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.022841 21010 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.025480 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:20.026757 21010 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:58:20.027930 20649 leader_election.cc:343] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.023677 21047 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:58:20.038806 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 55844.3us
I20250116 02:58:20.040549 21047 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.030378 21104 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.044718 21104 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.046947 21110 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.047526 21110 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.049283 20798 tablet_bootstrap.cc:654] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.049623 21110 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.055640 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:20.057121 20649 leader_election.cc:343] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.058423 21104 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.060575 21047 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:20.063997 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 69724.8us
I20250116 02:58:20.065714 20799 tablet_bootstrap.cc:492] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:20.066426 21108 consensus_queue.cc:1035] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:20.071357 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:20.075306 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:20.075960 20605 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:20.080336 20571 leader_election.cc:343] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.081138 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:20.078769 21047 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:58:20.087167 20723 leader_election.cc:343] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:20.088805 21150 log.cc:926] Time spent T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20250116 02:58:20.102556 21151 consensus_queue.cc:1035] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20250116 02:58:20.111451 20605 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:20.161324 20723 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:20.162822 21047 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:20.163383 21047 raft_consensus.cc:695] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:20.163926 20799 tablet_bootstrap.cc:654] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.164315 21047 consensus_queue.cc:237] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.166739 21109 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:20.197770 20798 tablet_bootstrap.cc:492] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:20.198377 20798 ts_tablet_manager.cc:1397] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.262s user 0.015s sys 0.011s
I20250116 02:58:20.204871 20797 tablet_bootstrap.cc:492] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:20.205579 20797 ts_tablet_manager.cc:1397] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.325s user 0.016s sys 0.005s
I20250116 02:58:20.208410 20797 raft_consensus.cc:357] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.205097 20799 tablet_bootstrap.cc:492] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:20.210453 20799 ts_tablet_manager.cc:1397] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.145s user 0.009s sys 0.010s
I20250116 02:58:20.234359 20799 raft_consensus.cc:357] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.216213 20798 raft_consensus.cc:357] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.247634 20799 raft_consensus.cc:383] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.247990 20798 raft_consensus.cc:383] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.248134 20799 raft_consensus.cc:738] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.209131 20797 raft_consensus.cc:383] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.249891 20797 raft_consensus.cc:738] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.248339 20798 raft_consensus.cc:738] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.251224 20798 consensus_queue.cc:260] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:20.244751 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 247932us
W20250116 02:58:20.252907 20720 leader_election.cc:343] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.254103 20797 consensus_queue.cc:260] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.259627 20798 ts_tablet_manager.cc:1428] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
I20250116 02:58:20.260896 20798 tablet_bootstrap.cc:492] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:20.249082 20799 consensus_queue.cc:260] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.265952 20799 ts_tablet_manager.cc:1428] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
I20250116 02:58:20.267378 20799 tablet_bootstrap.cc:492] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:20.274180 20647 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:20.275808 21114 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:20.276281 21114 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:20.276614 21114 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.257714 20797 ts_tablet_manager.cc:1428] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.052s user 0.005s sys 0.001s
I20250116 02:58:20.283452 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:20.284098 20797 tablet_bootstrap.cc:492] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:20.284971 21114 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.286257 20677 raft_consensus.cc:2388] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:20.290266 21114 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.291358 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:58:20.294595 20649 leader_election.cc:343] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.294958 20506 catalog_manager.cc:5526] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:20.297544 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:20.297951 20798 tablet_bootstrap.cc:654] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.282429 20799 tablet_bootstrap.cc:654] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.310827 20797 tablet_bootstrap.cc:654] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.314381 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:20.318511 21155 consensus_queue.cc:1035] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:20.332417 20797 tablet_bootstrap.cc:492] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:20.333117 20797 ts_tablet_manager.cc:1397] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.010s
I20250116 02:58:20.334287 20799 tablet_bootstrap.cc:492] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:20.334861 20799 ts_tablet_manager.cc:1397] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.068s user 0.020s sys 0.003s
I20250116 02:58:20.335206 20863 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.335738 20863 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.337720 20863 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.338135 20799 raft_consensus.cc:357] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.338927 20799 raft_consensus.cc:383] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.339211 20799 raft_consensus.cc:738] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.340008 20799 consensus_queue.cc:260] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.346238 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:20.348201 20679 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:20.349834 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:20.356108 20797 raft_consensus.cc:357] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.354118 20799 ts_tablet_manager.cc:1428] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250116 02:58:20.356959 20571 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:20.358539 20863 raft_consensus.cc:2798] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:20.358953 20863 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:20.359254 20863 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.360735 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:20.361471 20750 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:20.356097 20753 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.367056 20863 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.357007 20797 raft_consensus.cc:383] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.369365 20797 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.370023 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:20.370082 20797 consensus_queue.cc:260] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.370807 20681 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.373466 20799 tablet_bootstrap.cc:492] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:20.373342 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:58:20.378793 20647 leader_election.cc:343] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.374922 20753 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:20.379418 20647 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:20.382683 20798 tablet_bootstrap.cc:492] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:20.383221 20798 ts_tablet_manager.cc:1397] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.123s user 0.012s sys 0.010s
I20250116 02:58:20.384547 20720 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:20.385478 20572 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 02:58:20.386247 20720 leader_election.cc:343] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.387838 21004 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:20.388281 21004 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:20.388651 21004 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.389010 20798 raft_consensus.cc:357] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.390214 20798 raft_consensus.cc:383] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.391866 20798 raft_consensus.cc:738] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.392716 20798 consensus_queue.cc:260] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.386574 21154 consensus_queue.cc:1035] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250116 02:58:20.395594 20798 ts_tablet_manager.cc:1428] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20250116 02:58:20.396030 20797 ts_tablet_manager.cc:1428] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.063s user 0.006s sys 0.000s
I20250116 02:58:20.385608 21160 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:20.401201 20798 tablet_bootstrap.cc:492] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
W20250116 02:58:20.403977 20572 leader_election.cc:343] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.407096 20863 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.415627 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:58:20.417480 20572 leader_election.cc:343] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.420321 20797 tablet_bootstrap.cc:492] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:20.439163 20799 tablet_bootstrap.cc:654] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.379402 20681 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:20.447820 20571 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:20.449306 21108 raft_consensus.cc:2798] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
W20250116 02:58:20.449710 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:20.449859 21108 raft_consensus.cc:695] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:20.451344 21140 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:20.452811 20647 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:20.454407 20810 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:20.454912 20810 raft_consensus.cc:695] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:20.457242 21108 consensus_queue.cc:237] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.455683 20810 consensus_queue.cc:237] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.458851 21004 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.462090 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:20.464483 20681 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.465505 21004 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.465699 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:20.466409 20753 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.480150 21159 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.480785 21159 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.487105 21159 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:20.493150 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:20.496008 20723 leader_election.cc:343] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.501549 20798 tablet_bootstrap.cc:654] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.508136 20797 tablet_bootstrap.cc:654] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.526921 20506 catalog_manager.cc:5526] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:20.541432 20681 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:20.531293 20504 catalog_manager.cc:5526] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:20.527807 20753 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:20.546850 20571 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:20.548084 21108 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:20.548589 21108 raft_consensus.cc:695] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:20.549443 21108 consensus_queue.cc:237] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.556488 20799 tablet_bootstrap.cc:492] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:20.557305 20799 ts_tablet_manager.cc:1397] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.187s user 0.018s sys 0.006s
I20250116 02:58:20.511787 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:20.578552 20799 raft_consensus.cc:357] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.579738 20799 raft_consensus.cc:383] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.580031 20799 raft_consensus.cc:738] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
W20250116 02:58:20.580996 20720 leader_election.cc:343] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.581647 20720 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:20.581218 20799 consensus_queue.cc:260] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.584681 20868 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:20.590214 20799 ts_tablet_manager.cc:1428] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.032s user 0.002s sys 0.005s
I20250116 02:58:20.596359 20797 tablet_bootstrap.cc:492] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:20.596819 20797 ts_tablet_manager.cc:1397] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.183s user 0.025s sys 0.037s
I20250116 02:58:20.599956 20799 tablet_bootstrap.cc:492] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:20.600778 20798 tablet_bootstrap.cc:492] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:20.601203 20798 ts_tablet_manager.cc:1397] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.200s user 0.017s sys 0.010s
I20250116 02:58:20.607429 20799 tablet_bootstrap.cc:654] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.608799 20798 raft_consensus.cc:357] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.609653 20798 raft_consensus.cc:383] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.609957 20798 raft_consensus.cc:738] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.610666 20798 consensus_queue.cc:260] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.613036 20798 ts_tablet_manager.cc:1428] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250116 02:58:20.613039 20506 catalog_manager.cc:5526] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:20.616307 20798 tablet_bootstrap.cc:492] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:20.616677 20797 raft_consensus.cc:357] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.617522 20797 raft_consensus.cc:383] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.617813 20797 raft_consensus.cc:738] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.618453 20797 consensus_queue.cc:260] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.622383 20797 ts_tablet_manager.cc:1428] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
W20250116 02:58:20.622798 20720 consensus_peers.cc:487] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f -> Peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Couldn't send request to peer 231d91433a404468a44ea0a107345aeb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:20.624853 20797 tablet_bootstrap.cc:492] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:20.626571 20798 tablet_bootstrap.cc:654] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.641065 20797 tablet_bootstrap.cc:654] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.646523 20799 tablet_bootstrap.cc:492] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:20.647119 20799 ts_tablet_manager.cc:1397] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.003s
I20250116 02:58:20.649533 20797 tablet_bootstrap.cc:492] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:20.650354 20798 tablet_bootstrap.cc:492] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:20.650192 20797 ts_tablet_manager.cc:1397] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.001s
I20250116 02:58:20.650064 20799 raft_consensus.cc:357] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.656682 20798 ts_tablet_manager.cc:1397] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.041s user 0.017s sys 0.013s
I20250116 02:58:20.656742 20799 raft_consensus.cc:383] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.657531 20799 raft_consensus.cc:738] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.658322 20799 consensus_queue.cc:260] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.660596 20799 ts_tablet_manager.cc:1428] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20250116 02:58:20.660324 20798 raft_consensus.cc:357] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.661114 20798 raft_consensus.cc:383] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.661375 20798 raft_consensus.cc:738] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.662032 20798 consensus_queue.cc:260] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.662853 20799 tablet_bootstrap.cc:492] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:20.664651 20798 ts_tablet_manager.cc:1428] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250116 02:58:20.665812 20798 tablet_bootstrap.cc:492] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:20.670331 20799 tablet_bootstrap.cc:654] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.672226 20798 tablet_bootstrap.cc:654] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.687196 20798 tablet_bootstrap.cc:492] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:20.687155 20799 tablet_bootstrap.cc:492] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:20.687917 20798 ts_tablet_manager.cc:1397] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.000s
I20250116 02:58:20.687999 20799 ts_tablet_manager.cc:1397] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.007s
I20250116 02:58:20.695211 20797 raft_consensus.cc:357] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.697448 21171 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.698683 21171 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.696965 20799 raft_consensus.cc:357] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.701488 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:20.701467 20798 raft_consensus.cc:357] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.702082 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:20.702332 20798 raft_consensus.cc:383] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.702804 20798 raft_consensus.cc:738] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.703008 20608 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
W20250116 02:58:20.703608 20647 leader_election.cc:343] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.701526 21171 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.704201 20649 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:20.701695 20799 raft_consensus.cc:383] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.697906 20797 raft_consensus.cc:383] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.705443 20939 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:20.705844 20939 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:20.705971 20799 raft_consensus.cc:738] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.706274 20939 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.703619 20798 consensus_queue.cc:260] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.709633 20798 ts_tablet_manager.cc:1428] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20250116 02:58:20.709710 20799 consensus_queue.cc:260] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.711160 20798 tablet_bootstrap.cc:492] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:20.712508 20799 ts_tablet_manager.cc:1428] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20250116 02:58:20.705807 20797 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.714221 20939 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.719904 20797 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.722846 20797 ts_tablet_manager.cc:1428] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.066s user 0.007s sys 0.001s
I20250116 02:58:20.723899 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:20.724673 20608 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.726965 20868 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.727648 20797 tablet_bootstrap.cc:492] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:20.727511 20868 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.717674 21159 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:20.718690 20798 tablet_bootstrap.cc:654] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.729712 20868 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:20.730434 21108 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.731021 21108 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.731585 20939 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.732779 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:20.733680 20677 raft_consensus.cc:2388] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:58:20.737074 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:20.737589 20797 tablet_bootstrap.cc:654] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.724714 20799 tablet_bootstrap.cc:492] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:20.738557 20608 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:20.740774 20649 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:20.741626 20939 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:20.742106 20939 raft_consensus.cc:695] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:20.742933 20939 consensus_queue.cc:237] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.745891 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:20.746903 20750 raft_consensus.cc:2388] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:20.747025 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:58:20.748697 20647 leader_election.cc:343] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.749855 20799 tablet_bootstrap.cc:654] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.754381 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:20.755251 20723 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:20.755986 20571 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:20.757006 21004 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.741658 21108 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.757004 21159 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.764675 20799 tablet_bootstrap.cc:492] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:20.765823 20799 ts_tablet_manager.cc:1397] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.007s
I20250116 02:58:20.768281 21110 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.768903 21110 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.771183 21110 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.773159 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:20.773365 21004 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:20.774060 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:20.776497 20647 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:20.777268 21103 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:20.784407 20797 tablet_bootstrap.cc:492] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:20.785013 20797 ts_tablet_manager.cc:1397] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.006s
I20250116 02:58:20.791944 21159 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:20.793552 20798 tablet_bootstrap.cc:492] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:20.799695 20798 ts_tablet_manager.cc:1397] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.012s
I20250116 02:58:20.807499 20798 raft_consensus.cc:357] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.808348 20798 raft_consensus.cc:383] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.808638 20798 raft_consensus.cc:738] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.809520 20798 consensus_queue.cc:260] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.812045 20798 ts_tablet_manager.cc:1428] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250116 02:58:20.768846 20799 raft_consensus.cc:357] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.812536 20799 raft_consensus.cc:383] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.812841 20799 raft_consensus.cc:738] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.813494 20799 consensus_queue.cc:260] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.814088 20798 tablet_bootstrap.cc:492] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:20.812089 20797 raft_consensus.cc:357] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.819684 20797 raft_consensus.cc:383] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.819078 20503 catalog_manager.cc:5526] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:20.819998 20797 raft_consensus.cc:738] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.821203 20797 consensus_queue.cc:260] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.830546 20798 tablet_bootstrap.cc:654] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.830935 20799 ts_tablet_manager.cc:1428] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.065s user 0.008s sys 0.005s
I20250116 02:58:20.836994 20799 tablet_bootstrap.cc:492] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:20.835808 20797 ts_tablet_manager.cc:1428] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.050s user 0.003s sys 0.009s
I20250116 02:58:20.838647 20797 tablet_bootstrap.cc:492] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:20.844419 20797 tablet_bootstrap.cc:654] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.848978 21010 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.849650 21010 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.852404 20799 tablet_bootstrap.cc:654] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.852985 21010 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:20.852859 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:20.854144 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:58:20.854523 20723 leader_election.cc:343] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:20.856609 20720 leader_election.cc:343] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.857112 20720 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:20.863396 21010 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:20.885499 20799 tablet_bootstrap.cc:492] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:20.886242 20799 ts_tablet_manager.cc:1397] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.000s
I20250116 02:58:20.886775 21154 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.887428 21154 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.890667 21154 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.890349 20799 raft_consensus.cc:357] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.891498 20799 raft_consensus.cc:383] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.891809 20799 raft_consensus.cc:738] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.892019 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:20.892333 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:20.892807 20749 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:20.894181 20572 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:58:20.894970 20571 leader_election.cc:343] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.895823 20797 tablet_bootstrap.cc:492] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:20.896359 20797 ts_tablet_manager.cc:1397] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.058s user 0.021s sys 0.026s
I20250116 02:58:20.897009 21154 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:20.897482 21154 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:20.897884 21154 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.892542 20799 consensus_queue.cc:260] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.900288 20797 raft_consensus.cc:357] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.901266 20797 raft_consensus.cc:383] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.901542 20797 raft_consensus.cc:738] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.902267 20797 consensus_queue.cc:260] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.906071 20797 ts_tablet_manager.cc:1428] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20250116 02:58:20.907377 20797 tablet_bootstrap.cc:492] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:20.911427 20798 tablet_bootstrap.cc:492] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:20.912144 20798 ts_tablet_manager.cc:1397] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.098s user 0.027s sys 0.044s
I20250116 02:58:20.914566 20797 tablet_bootstrap.cc:654] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.916939 21154 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.901854 20799 ts_tablet_manager.cc:1428] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20250116 02:58:20.917555 20798 raft_consensus.cc:357] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.918514 20798 raft_consensus.cc:383] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.919077 20798 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.919642 20799 tablet_bootstrap.cc:492] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:20.920562 21010 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.921041 21010 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.923254 21010 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:20.924746 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:20.919935 20798 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.926249 21110 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:20.926806 21110 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.927357 20798 ts_tablet_manager.cc:1428] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250116 02:58:20.929759 20799 tablet_bootstrap.cc:654] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.931785 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:20.933519 21110 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:58:20.934523 20647 leader_election.cc:343] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:20.935604 20723 leader_election.cc:343] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.934541 20798 tablet_bootstrap.cc:492] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:20.938076 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:20.938877 20677 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:20.939169 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:20.939905 20679 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.940831 20720 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:20.935101 21154 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:20.942101 21010 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:20.942476 21010 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:20.942829 21010 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.943389 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:20.944137 20749 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.947491 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:20.948467 20608 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:20.949930 20649 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:20.954660 20939 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:20.955114 20939 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:20.955436 20939 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.955524 21010 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.957748 21010 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:20.961400 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:20.962167 20681 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:20.970278 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:20.971865 20798 tablet_bootstrap.cc:654] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
W20250116 02:58:20.972741 20723 leader_election.cc:343] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:20.979290 20799 tablet_bootstrap.cc:492] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:20.980034 20799 ts_tablet_manager.cc:1397] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.009s
I20250116 02:58:20.983083 20799 raft_consensus.cc:357] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.983960 20799 raft_consensus.cc:383] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.984289 20799 raft_consensus.cc:738] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.985005 20799 consensus_queue.cc:260] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.987903 20799 ts_tablet_manager.cc:1428] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250116 02:58:20.989187 20799 tablet_bootstrap.cc:492] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:20.990077 20749 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:20.990484 20798 tablet_bootstrap.cc:492] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:20.990993 20798 ts_tablet_manager.cc:1397] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.000s
I20250116 02:58:20.970376 20939 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.994084 20798 raft_consensus.cc:357] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:20.997153 20798 raft_consensus.cc:383] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:20.997448 20798 raft_consensus.cc:738] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:20.997997 20799 tablet_bootstrap.cc:654] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.998121 20798 consensus_queue.cc:260] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.000735 20798 ts_tablet_manager.cc:1428] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250116 02:58:21.001055 20939 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.001824 20798 tablet_bootstrap.cc:492] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:21.003376 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:21.004153 20605 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.008553 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:58:21.012296 20647 leader_election.cc:343] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:21.008453 20798 tablet_bootstrap.cc:654] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:20.996477 20679 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:21.028892 20798 tablet_bootstrap.cc:492] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:21.029449 20798 ts_tablet_manager.cc:1397] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.006s
I20250116 02:58:21.032366 20798 raft_consensus.cc:357] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.033180 20798 raft_consensus.cc:383] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:21.033488 20798 raft_consensus.cc:738] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:21.034209 20798 consensus_queue.cc:260] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.037482 21154 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:21.042675 20798 ts_tablet_manager.cc:1428] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250116 02:58:21.043512 21171 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:21.043954 20798 tablet_bootstrap.cc:492] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:21.044128 21171 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.045333 21159 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:21.045807 21159 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.046468 21171 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.047827 21159 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:21.048971 20797 tablet_bootstrap.cc:492] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:21.049525 20797 ts_tablet_manager.cc:1397] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.142s user 0.031s sys 0.070s
I20250116 02:58:21.051702 20798 tablet_bootstrap.cc:654] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:21.068876 20798 tablet_bootstrap.cc:492] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:21.069459 20798 ts_tablet_manager.cc:1397] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.012s
I20250116 02:58:21.072748 20798 raft_consensus.cc:357] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.073561 20798 raft_consensus.cc:383] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:21.073858 20798 raft_consensus.cc:738] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:21.074534 20798 consensus_queue.cc:260] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.078040 20798 ts_tablet_manager.cc:1428] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250116 02:58:21.079288 20798 tablet_bootstrap.cc:492] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Bootstrap starting.
I20250116 02:58:21.085978 20798 tablet_bootstrap.cc:654] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:21.087545 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:21.089206 20680 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:21.092494 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:21.073266 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:21.102576 20681 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
W20250116 02:58:21.103621 20649 leader_election.cc:343] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:21.107496 20797 raft_consensus.cc:357] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.108448 20797 raft_consensus.cc:383] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:21.108721 20797 raft_consensus.cc:738] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:21.109452 20797 consensus_queue.cc:260] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.112054 20797 ts_tablet_manager.cc:1428] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20250116 02:58:21.107405 21108 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:21.113195 21108 raft_consensus.cc:695] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:21.114303 21108 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.117166 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:21.116402 20797 tablet_bootstrap.cc:492] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:21.132293 21103 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:21.140771 20797 tablet_bootstrap.cc:654] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:21.148555 21103 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:21.155388 20720 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
W20250116 02:58:21.161351 20723 leader_election.cc:343] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:21.169865 20868 raft_consensus.cc:2798] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:21.170313 20868 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:21.170643 20868 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.183326 21171 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:21.184888 20863 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:21.196878 20868 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.233629 20868 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:21.237016 20799 tablet_bootstrap.cc:492] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:21.237588 20799 ts_tablet_manager.cc:1397] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.249s user 0.100s sys 0.106s
I20250116 02:58:21.237877 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:21.203840 20798 tablet_bootstrap.cc:492] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: No bootstrap required, opened a new log
I20250116 02:58:21.209357 21157 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:58:21.216410 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 61145.2us
I20250116 02:58:21.220005 21154 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:21.240636 21157 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.240797 20799 raft_consensus.cc:357] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.223006 20797 tablet_bootstrap.cc:492] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:21.241509 20799 raft_consensus.cc:383] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:21.241802 20799 raft_consensus.cc:738] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:21.241968 20797 ts_tablet_manager.cc:1397] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.126s user 0.014s sys 0.002s
I20250116 02:58:21.242529 20799 consensus_queue.cc:260] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.243518 21157 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.245101 20799 ts_tablet_manager.cc:1428] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250116 02:58:21.245152 20797 raft_consensus.cc:357] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.223886 21171 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.248600 20797 raft_consensus.cc:383] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:21.248816 20798 ts_tablet_manager.cc:1397] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Time spent bootstrapping tablet: real 0.170s user 0.015s sys 0.036s
I20250116 02:58:21.249037 20797 raft_consensus.cc:738] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:21.239490 20503 catalog_manager.cc:5526] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:21.250419 20720 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:21.251763 20798 raft_consensus.cc:357] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.252532 20798 raft_consensus.cc:383] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:21.252426 21159 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:21.252868 20798 raft_consensus.cc:738] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Initialized, Role: FOLLOWER
I20250116 02:58:21.253618 20798 consensus_queue.cc:260] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.255187 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:21.246361 20799 tablet_bootstrap.cc:492] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:21.257615 21171 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.251267 20797 consensus_queue.cc:260] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.253156 21159 raft_consensus.cc:695] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:21.259479 21159 consensus_queue.cc:237] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:21.260062 20723 leader_election.cc:343] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:21.262897 20798 ts_tablet_manager.cc:1428] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250116 02:58:21.263059 20799 tablet_bootstrap.cc:654] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:21.264662 20606 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:21.264837 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:21.265578 20681 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:21.225200 20605 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:21.266935 20649 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:21.267874 20939 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:21.268050 21157 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:21.268333 20939 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:21.268707 20939 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.268673 21157 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.270506 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:21.272248 21157 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.272912 21157 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:21.273274 21157 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:21.273623 21157 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.275465 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:21.276355 20681 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
I20250116 02:58:21.278523 21103 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:21.279147 21103 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.225086 21108 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:21.283748 20571 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:21.284312 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:21.284701 21154 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:21.285267 21154 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:21.285701 21154 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:21.285799 20649 leader_election.cc:343] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:21.289011 20939 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.291929 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:21.292610 20606 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.295020 21103 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.298466 20939 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.300164 20606 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:21.302501 20797 ts_tablet_manager.cc:1428] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.060s user 0.002s sys 0.005s
I20250116 02:58:21.303982 20649 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:58:21.305524 21103 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:21.306727 21151 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:21.307353 21151 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.307852 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:21.308763 20751 raft_consensus.cc:2388] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:58:21.311719 20797 tablet_bootstrap.cc:492] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:21.316210 21151 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:21.320780 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:21.321709 20608 raft_consensus.cc:2388] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:21.324492 21159 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:21.325068 21159 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.327898 21159 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:21.329958 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:58:21.331549 20723 leader_election.cc:343] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:21.331986 21157 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.333159 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:21.333921 20751 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:21.334124 21157 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.335844 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:21.336616 20750 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
I20250116 02:58:21.340328 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:21.341176 20677 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.342819 21103 raft_consensus.cc:695] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:21.343998 20799 tablet_bootstrap.cc:492] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:21.346261 20649 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:21.321971 21154 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.347177 20939 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:21.350243 20677 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:21.350685 20939 raft_consensus.cc:695] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:21.351603 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:21.351544 20939 consensus_queue.cc:237] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.352445 21108 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:21.352945 21108 raft_consensus.cc:695] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:21.353770 21108 consensus_queue.cc:237] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.346930 20506 catalog_manager.cc:5526] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:21.347786 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:21.358378 20750 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.348191 20799 ts_tablet_manager.cc:1397] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.102s user 0.034s sys 0.032s
I20250116 02:58:21.368371 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:21.348086 21103 consensus_queue.cc:237] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.369115 20677 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.366408 20799 raft_consensus.cc:357] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.372231 20799 raft_consensus.cc:383] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:21.370417 20647 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:21.374281 21110 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.374935 20799 raft_consensus.cc:738] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:21.381052 20799 consensus_queue.cc:260] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.385295 20750 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:21.387063 20503 catalog_manager.cc:5526] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:21.388351 21171 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:21.388548 20677 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:58:21.399063 20571 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:21.400141 20863 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:21.400764 20863 raft_consensus.cc:695] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:21.401599 20863 consensus_queue.cc:237] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.397325 20680 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.409340 20799 ts_tablet_manager.cc:1428] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.046s user 0.007s sys 0.001s
I20250116 02:58:21.401050 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:21.410416 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:21.411182 20682 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 0.
I20250116 02:58:21.396634 21171 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.415004 21171 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.420058 20797 tablet_bootstrap.cc:654] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:21.400115 21154 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:21.424856 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:21.435274 20799 tablet_bootstrap.cc:492] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f: Bootstrap starting.
I20250116 02:58:21.440085 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:21.444886 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:21.454308 20720 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:58:21.457667 20572 leader_election.cc:343] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:21.454285 20504 catalog_manager.cc:5526] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:21.465937 20504 catalog_manager.cc:5526] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:21.476477 20720 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:21.479840 20868 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.488050 21151 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:21.488579 21151 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:21.488950 21151 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.489634 20868 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:21.444890 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:21.456997 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:21.556823 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:21.558568 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:21.565426 20753 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 0.
W20250116 02:58:21.567958 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 69965.4us
I20250116 02:58:21.452343 21171 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.108s
W20250116 02:58:21.578774 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 86343.4us
W20250116 02:58:21.579901 20572 leader_election.cc:343] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:21.569705 20754 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 0.
W20250116 02:58:21.617709 20647 consensus_peers.cc:487] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb -> Peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Couldn't send request to peer 17446c8dacf04461acb843547665250f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:21.642999 20647 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:58:21.650925 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 163112us
W20250116 02:58:21.689025 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:21.760366 20799 tablet_bootstrap.cc:654] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:21.795641 20647 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:21.797888 21171 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:58:21.798754 21171 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:21.799522 21171 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.756520 21219 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250116 02:58:21.808557 21221 log.cc:926] Time spent T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.083s user 0.004s sys 0.006s
I20250116 02:58:21.808128 21219 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:58:21.823210 20799 tablet_bootstrap.cc:492] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f: No bootstrap required, opened a new log
I20250116 02:58:21.823933 20799 ts_tablet_manager.cc:1397] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f: Time spent bootstrapping tablet: real 0.398s user 0.009s sys 0.011s
I20250116 02:58:21.837909 20504 catalog_manager.cc:5526] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:21.822304 21219 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:21.858383 21159 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:58:21.938505 21110 consensus_meta.cc:287] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.564s user 0.009s sys 0.002s
I20250116 02:58:21.958644 21159 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:21.975355 21110 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250116 02:58:22.054060 21151 consensus_meta.cc:287] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.565s user 0.009s sys 0.003s
I20250116 02:58:22.080837 21151 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.091812 21159 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:22.060367 21226 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:22.108603 21226 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.111030 21226 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:22.079169 21171 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.112991 21171 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:58:22.055881 20649 leader_election.cc:343] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
W20250116 02:58:22.115130 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 59291.2us
W20250116 02:58:22.097046 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 284726us
I20250116 02:58:22.123782 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:22.124680 20683 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:22.095299 20799 raft_consensus.cc:357] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.128162 20799 raft_consensus.cc:383] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:22.128566 20799 raft_consensus.cc:738] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Initialized, Role: FOLLOWER
I20250116 02:58:22.129383 20799 consensus_queue.cc:260] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.130132 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:22.131510 20799 ts_tablet_manager.cc:1428] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f: Time spent starting tablet: real 0.307s user 0.005s sys 0.005s
I20250116 02:58:22.107782 21151 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:22.092108 20797 tablet_bootstrap.cc:492] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:22.140740 20797 ts_tablet_manager.cc:1397] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.829s user 0.042s sys 0.067s
I20250116 02:58:22.143776 20797 raft_consensus.cc:357] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.144709 20797 raft_consensus.cc:383] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:22.145749 20797 raft_consensus.cc:738] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:22.148041 20797 consensus_queue.cc:260] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:22.140108 21227 log.cc:926] Time spent T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.342s user 0.005s sys 0.009s
I20250116 02:58:22.154112 21004 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:22.193141 20797 ts_tablet_manager.cc:1428] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.052s user 0.005s sys 0.001s
W20250116 02:58:22.156440 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 52008.5us
I20250116 02:58:22.159524 21157 consensus_queue.cc:1035] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:22.161401 21215 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:22.106005 21219 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.179726 21225 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:22.195875 20797 tablet_bootstrap.cc:492] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:22.202962 20682 raft_consensus.cc:2388] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:22.204008 20797 tablet_bootstrap.cc:654] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:22.185230 20720 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:22.186661 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:22.205487 21225 raft_consensus.cc:2798] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:22.194872 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:22.209973 21225 raft_consensus.cc:695] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:22.211745 21225 consensus_queue.cc:237] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.216547 20755 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:22.219206 20756 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:22.221660 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:22.223577 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:22.224531 20606 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:22.275094 20606 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:58:22.315443 21219 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:22.316921 21219 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250116 02:58:22.319705 20649 leader_election.cc:343] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Illegal state: must be running to vote when last-logged opid is not known
I20250116 02:58:22.323084 21110 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:22.327538 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:22.328119 20606 raft_consensus.cc:2388] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:58:22.331920 21160 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:22.331648 21140 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:22.336978 20797 tablet_bootstrap.cc:492] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:22.334228 21160 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.338699 20797 ts_tablet_manager.cc:1397] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.143s user 0.019s sys 0.000s
I20250116 02:58:22.331920 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:22.345302 21140 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:22.349788 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:22.361333 20750 raft_consensus.cc:2388] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 1: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:58:22.395781 20647 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:22.380311 20753 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:22.415094 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 79075.4us
I20250116 02:58:22.432016 20723 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:22.433408 20797 raft_consensus.cc:357] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.434245 20797 raft_consensus.cc:383] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:22.434552 20797 raft_consensus.cc:738] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
W20250116 02:58:22.437198 21246 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
I20250116 02:58:22.437711 20797 consensus_queue.cc:260] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.415778 21226 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250116 02:58:22.451115 20797 ts_tablet_manager.cc:1428] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.112s user 0.007s sys 0.000s
I20250116 02:58:22.459996 20797 tablet_bootstrap.cc:492] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:22.464843 21249 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:22.516521 21216 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:22.517196 21216 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.520188 21216 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:22.490432 20753 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:58:22.534829 20797 tablet_bootstrap.cc:654] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:22.540402 21207 consensus_queue.cc:1035] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:22.540911 21216 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:22.541932 21207 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250116 02:58:22.553359 20723 consensus_peers.cc:487] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f -> Peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Couldn't send request to peer f65e8d7f76144fab9725ca32a6328c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250116 02:58:22.563292 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 129579us
I20250116 02:58:22.564527 20723 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:58:22.565981 21249 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:22.566644 21249 raft_consensus.cc:695] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:22.567677 21249 consensus_queue.cc:237] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.573161 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:22.582576 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:22.595927 20680 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:22.585935 20751 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:22.580395 20797 tablet_bootstrap.cc:492] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:22.607334 20797 ts_tablet_manager.cc:1397] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.148s user 0.008s sys 0.007s
I20250116 02:58:22.610970 20647 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:22.614868 20797 raft_consensus.cc:357] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.618000 21160 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:22.619097 20797 raft_consensus.cc:383] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:22.619503 20797 raft_consensus.cc:738] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:22.621438 20797 consensus_queue.cc:260] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.623323 20797 ts_tablet_manager.cc:1428] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250116 02:58:22.623443 21140 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:58:22.623916 21140 raft_consensus.cc:695] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:22.624917 20797 tablet_bootstrap.cc:492] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f: Bootstrap starting.
I20250116 02:58:22.619805 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:58:22.624750 21140 consensus_queue.cc:237] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.653713 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:22.654661 20605 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:58:22.654853 20751 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 1: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:58:22.655880 20504 catalog_manager.cc:5526] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:22.675046 20647 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:22.694728 21219 raft_consensus.cc:2743] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:58:22.717079 20572 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:22.720665 21004 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:22.731124 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 64563us
I20250116 02:58:22.754972 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:22.774379 20797 tablet_bootstrap.cc:654] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f: Neither blocks nor log segments found. Creating new log.
I20250116 02:58:22.785893 20720 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:58:22.790030 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 52126.3us
I20250116 02:58:22.800447 21004 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250116 02:58:22.824440 20723 consensus_peers.cc:487] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f -> Peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Couldn't send request to peer f65e8d7f76144fab9725ca32a6328c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:22.828281 21010 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:58:22.840845 20797 tablet_bootstrap.cc:492] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f: No bootstrap required, opened a new log
I20250116 02:58:22.841341 20797 ts_tablet_manager.cc:1397] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f: Time spent bootstrapping tablet: real 0.217s user 0.020s sys 0.025s
I20250116 02:58:22.843719 20797 raft_consensus.cc:357] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.844460 20797 raft_consensus.cc:383] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:58:22.844259 21272 consensus_queue.cc:1035] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:22.844736 20797 raft_consensus.cc:738] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Initialized, Role: FOLLOWER
I20250116 02:58:22.845513 20797 consensus_queue.cc:260] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.852511 20797 ts_tablet_manager.cc:1428] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250116 02:58:22.862486 20504 catalog_manager.cc:5526] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
W20250116 02:58:22.877559 20554 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250116 02:58:22.904109 20504 catalog_manager.cc:5526] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:22.986794 21108 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:22.987385 21108 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:22.990636 21108 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:22.991246 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:58:22.993485 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:58:22.994823 20572 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:58:22.995455 21108 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250116 02:58:23.013434 21219 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:23.046828 21236 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:23.130709 21267 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250116 02:58:23.144634 21010 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:58:23.303889 20605 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 0 FOLLOWER]: Advancing to term 1
W20250116 02:58:23.714550 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 45456.8us
I20250116 02:58:24.093505 19386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:58:24.134166 21293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:58:24.135190 21291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:58:24.135901 21290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:58:24.137406 19386 server_base.cc:1034] running on GCE node
I20250116 02:58:24.138391 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250116 02:58:24.138640 19386 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250116 02:58:24.138808 19386 hybrid_clock.cc:648] HybridClock initialized: now 1736996304138791 us; error 0 us; skew 500 ppm
I20250116 02:58:24.139345 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:58:24.146559 19386 webserver.cc:458] Webserver started at http://127.18.238.132:40841/ using document root <none> and password file <none>
I20250116 02:58:24.147257 19386 fs_manager.cc:362] Metadata directory not provided
I20250116 02:58:24.147503 19386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:58:24.148002 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:58:24.149569 19386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-3-root/instance:
uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
format_stamp: "Formatted at 2025-01-16 02:58:24 on dist-test-slave-qn1x"
I20250116 02:58:24.155416 19386 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20250116 02:58:24.160583 21298 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:24.161409 19386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250116 02:58:24.161718 19386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-3-root
uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
format_stamp: "Formatted at 2025-01-16 02:58:24 on dist-test-slave-qn1x"
I20250116 02:58:24.162056 19386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:58:24.180500 19386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:58:24.182217 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:58:24.183951 19386 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:58:24.186800 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:58:24.187088 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:24.187341 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:58:24.187528 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:58:24.257086 19386 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.132:33959
I20250116 02:58:24.257181 21360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.132:33959 every 8 connection(s)
I20250116 02:58:24.270335 19386 auto_leader_rebalancer-test.cc:413] add a tserver: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:24.293466 21361 heartbeater.cc:346] Connected to a master server at 127.18.238.190:37551
I20250116 02:58:24.294013 21361 heartbeater.cc:463] Registering TS with master...
I20250116 02:58:24.294950 21361 heartbeater.cc:509] Master 127.18.238.190:37551 requested a full tablet report, sending...
I20250116 02:58:24.297111 20506 ts_manager.cc:194] Registered new tserver with Master: 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:24.298871 20506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48594
I20250116 02:58:25.011108 20605 consensus_queue.cc:237] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.020520 20678 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:25.023509 21215 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:25.027753 20753 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:25.035683 20571 consensus_peers.cc:487] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 221670297d85466bac4e5a9bfead9b18. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:25.040138 21215 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:25.054778 20750 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.052624 21374 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.057476 20678 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.077165 20506 catalog_manager.cc:5526] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.119513 20676 consensus_queue.cc:237] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.130743 20752 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:25.132988 21222 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:25.134922 20605 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:25.137893 21222 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250116 02:58:25.150269 20646 consensus_peers.cc:487] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d7658a3a42c44f485085e75a4bd7b88. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:25.152423 21284 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.155088 20603 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.159461 20752 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.170639 20676 consensus_queue.cc:237] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.177982 20504 catalog_manager.cc:5526] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:25.194630 20646 consensus_peers.cc:487] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bddd1f65156348fcb619edd31a8971f8. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:25.196870 20752 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:25.197014 20605 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:25.198590 21284 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:25.202252 21222 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:25.211344 21284 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.219194 20750 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.235519 20605 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.240435 20503 catalog_manager.cc:5526] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:25.242565 20753 consensus_queue.cc:237] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.264923 20603 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:25.267971 20676 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:25.269763 21271 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:25.271014 21267 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250116 02:58:25.278342 20720 consensus_peers.cc:487] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49f9e38171174b70b4373b56dbcd90be. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:25.287128 21271 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.288971 20682 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.293666 20603 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.304859 21361 heartbeater.cc:501] Master 127.18.238.190:37551 was elected leader, sending a full tablet report...
I20250116 02:58:25.309167 20504 catalog_manager.cc:5526] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:25.602469 21394 ts_tablet_manager.cc:927] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:25.605041 21394 tablet_copy_client.cc:323] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:25.636513 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet 221670297d85466bac4e5a9bfead9b18 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:25.637238 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet 221670297d85466bac4e5a9bfead9b18 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-221670297d85466bac4e5a9bfead9b18
I20250116 02:58:25.644815 20618 tablet_copy_source_session.cc:215] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:25.654886 21394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221670297d85466bac4e5a9bfead9b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:25.677917 21394 tablet_copy_client.cc:806] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:25.678583 21394 tablet_copy_client.cc:670] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:25.689702 21394 tablet_copy_client.cc:538] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:25.708606 21394 tablet_bootstrap.cc:492] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:25.734524 21394 tablet_bootstrap.cc:492] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:25.736907 21394 tablet_bootstrap.cc:492] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:25.737851 21394 ts_tablet_manager.cc:1397] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.000s
I20250116 02:58:25.743382 21394 raft_consensus.cc:357] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.744628 21394 raft_consensus.cc:738] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:25.746462 21394 consensus_queue.cc:260] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.751912 21394 ts_tablet_manager.cc:1428] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20250116 02:58:25.753886 21399 ts_tablet_manager.cc:927] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:25.757972 21399 tablet_copy_client.cc:323] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:25.759442 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-221670297d85466bac4e5a9bfead9b18 received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:25.760057 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-221670297d85466bac4e5a9bfead9b18 on tablet 221670297d85466bac4e5a9bfead9b18 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:25.769441 21394 ts_tablet_manager.cc:927] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:25.776423 21394 tablet_copy_client.cc:323] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:25.778760 20692 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet bddd1f65156348fcb619edd31a8971f8 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:25.779645 20692 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet bddd1f65156348fcb619edd31a8971f8 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-bddd1f65156348fcb619edd31a8971f8
I20250116 02:58:25.778760 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 1d7658a3a42c44f485085e75a4bd7b88 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:25.786931 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 1d7658a3a42c44f485085e75a4bd7b88 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-1d7658a3a42c44f485085e75a4bd7b88
I20250116 02:58:25.787689 20692 tablet_copy_source_session.cc:215] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:25.791795 21394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddd1f65156348fcb619edd31a8971f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:25.801215 20693 tablet_copy_source_session.cc:215] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:25.804293 21399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7658a3a42c44f485085e75a4bd7b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:25.810967 21394 tablet_copy_client.cc:806] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:25.811723 21394 tablet_copy_client.cc:670] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:25.816118 21394 tablet_copy_client.cc:538] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:25.824869 21394 tablet_bootstrap.cc:492] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:25.825542 21399 tablet_copy_client.cc:806] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:25.826249 21399 tablet_copy_client.cc:670] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:25.829981 21399 tablet_copy_client.cc:538] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:25.843123 21399 tablet_bootstrap.cc:492] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:25.849359 21394 tablet_bootstrap.cc:492] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:25.851238 21394 tablet_bootstrap.cc:492] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:25.851976 21394 ts_tablet_manager.cc:1397] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.017s
I20250116 02:58:25.855131 21394 raft_consensus.cc:357] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.855999 21394 raft_consensus.cc:738] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:25.856582 21394 consensus_queue.cc:260] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.858842 21394 ts_tablet_manager.cc:1428] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250116 02:58:25.861199 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-bddd1f65156348fcb619edd31a8971f8 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:25.861692 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-bddd1f65156348fcb619edd31a8971f8 on tablet bddd1f65156348fcb619edd31a8971f8 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:25.871611 21394 ts_tablet_manager.cc:927] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:25.877385 21399 tablet_bootstrap.cc:492] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:25.878485 21399 tablet_bootstrap.cc:492] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:25.879446 21399 ts_tablet_manager.cc:1397] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.037s user 0.017s sys 0.011s
I20250116 02:58:25.883580 21399 raft_consensus.cc:357] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.884766 21399 raft_consensus.cc:738] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:25.885555 21399 consensus_queue.cc:260] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.887421 21394 tablet_copy_client.cc:323] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.131:43647
I20250116 02:58:25.890897 21399 ts_tablet_manager.cc:1428] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20250116 02:58:25.897454 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-1d7658a3a42c44f485085e75a4bd7b88 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:25.898098 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-1d7658a3a42c44f485085e75a4bd7b88 on tablet 1d7658a3a42c44f485085e75a4bd7b88 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:25.920500 20767 tablet_copy_service.cc:140] P 17446c8dacf04461acb843547665250f: Received BeginTabletCopySession request for tablet 49f9e38171174b70b4373b56dbcd90be from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:35540)
I20250116 02:58:25.921058 20767 tablet_copy_service.cc:161] P 17446c8dacf04461acb843547665250f: Beginning new tablet copy session on tablet 49f9e38171174b70b4373b56dbcd90be from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:35540: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-49f9e38171174b70b4373b56dbcd90be
I20250116 02:58:25.929548 20767 tablet_copy_source_session.cc:215] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:25.936859 21394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f9e38171174b70b4373b56dbcd90be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:25.956634 21394 tablet_copy_client.cc:806] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:25.957690 21394 tablet_copy_client.cc:670] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:25.961808 21394 tablet_copy_client.cc:538] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:25.972137 21394 tablet_bootstrap.cc:492] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:25.989533 21394 tablet_bootstrap.cc:492] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:25.990490 21394 tablet_bootstrap.cc:492] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:25.991135 21394 ts_tablet_manager.cc:1397] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20250116 02:58:25.993765 21394 raft_consensus.cc:357] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.994763 21394 raft_consensus.cc:738] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:25.995432 21394 consensus_queue.cc:260] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:25.997722 21394 ts_tablet_manager.cc:1428] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250116 02:58:25.999266 20767 tablet_copy_service.cc:342] P 17446c8dacf04461acb843547665250f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-49f9e38171174b70b4373b56dbcd90be received from {username='slave'} at 127.0.0.1:35540
I20250116 02:58:25.999758 20767 tablet_copy_service.cc:434] P 17446c8dacf04461acb843547665250f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-49f9e38171174b70b4373b56dbcd90be on tablet 49f9e38171174b70b4373b56dbcd90be with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:26.026772 21336 raft_consensus.cc:1212] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:26.199476 21336 raft_consensus.cc:1212] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:26.294009 21336 raft_consensus.cc:1212] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:26.349061 21336 raft_consensus.cc:1212] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:26.497413 21154 raft_consensus.cc:1059] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:26.501801 21154 consensus_queue.cc:237] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:26.517159 21336 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.518692 21412 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:26.524032 20750 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.527688 20677 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.530226 21154 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:26.531415 21215 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:26.544236 21215 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.546315 20682 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.549847 21336 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.560835 20753 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.599881 20504 catalog_manager.cc:5526] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.639775 21284 raft_consensus.cc:1059] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:26.641875 21284 consensus_queue.cc:237] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:26.646265 20604 consensus_queue.cc:237] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:26.651780 21336 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:26.652360 20752 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:26.653800 21215 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:26.654737 21412 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:26.659406 21335 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.659890 20603 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.663132 21410 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250116 02:58:26.665316 20752 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.666247 21373 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:26.667490 21215 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.673889 21335 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.679899 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 221670297d85466bac4e5a9bfead9b18 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:26.672472 21373 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20250116 02:58:26.677960 20752 raft_consensus.cc:2949] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.694782 20505 catalog_manager.cc:5526] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250116 02:58:26.705926 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 44510.5us
I20250116 02:58:26.745893 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet 221670297d85466bac4e5a9bfead9b18 with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:26.748620 21284 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.750749 20752 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.763835 21336 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.773581 20603 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.774443 20505 catalog_manager.cc:5526] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:26.787415 21428 tablet_replica.cc:331] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:26.817649 21428 raft_consensus.cc:2238] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:26.828842 20677 consensus_queue.cc:237] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:26.818758 21428 raft_consensus.cc:2267] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:26.844769 20603 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:26.847298 21284 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:26.845404 21336 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:26.853736 21428 ts_tablet_manager.cc:1905] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:26.861994 21409 raft_consensus.cc:1059] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:26.865103 21409 consensus_queue.cc:237] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:26.867826 21423 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:26.871665 21423 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.877282 21428 ts_tablet_manager.cc:1918] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:26.877904 21428 log.cc:1198] T 221670297d85466bac4e5a9bfead9b18 P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/221670297d85466bac4e5a9bfead9b18
I20250116 02:58:26.878958 21410 raft_consensus.cc:1059] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:26.883477 21336 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.883507 21410 consensus_queue.cc:237] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:26.887195 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet 221670297d85466bac4e5a9bfead9b18 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:26.892794 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet bddd1f65156348fcb619edd31a8971f8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:26.895874 21335 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.897483 20603 raft_consensus.cc:2949] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:26.899889 20604 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.901989 20503 catalog_manager.cc:5526] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:26.904356 21409 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:26.906131 20682 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.912333 21409 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:26.913419 21366 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:26.921030 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet bddd1f65156348fcb619edd31a8971f8 with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:26.987025 20606 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.992816 21335 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:26.993804 21373 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:26.997790 20753 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:27.002821 21373 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:27.013742 21424 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:27.021955 21430 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.023582 20606 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.050127 21434 tablet_replica.cc:331] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:27.051712 21434 raft_consensus.cc:2238] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:27.051785 21366 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.052851 21434 raft_consensus.cc:2267] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:27.060993 20682 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.072813 21335 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.077330 20503 catalog_manager.cc:5526] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:27.087393 21434 ts_tablet_manager.cc:1905] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:27.132330 21336 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.141387 21434 ts_tablet_manager.cc:1918] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:27.142110 21434 log.cc:1198] T bddd1f65156348fcb619edd31a8971f8 P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/bddd1f65156348fcb619edd31a8971f8
I20250116 02:58:27.144222 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet bddd1f65156348fcb619edd31a8971f8 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:27.115427 20606 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.128067 20503 catalog_manager.cc:5526] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:27.141577 20750 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.190070 20677 consensus_queue.cc:237] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:27.196038 21336 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:27.197856 21410 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:27.208135 21336 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.206877 21410 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.227913 20505 catalog_manager.cc:5526] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.231524 20751 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:27.235663 21373 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:27.236013 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 1d7658a3a42c44f485085e75a4bd7b88 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:27.251017 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7658a3a42c44f485085e75a4bd7b88 with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:27.253870 20751 raft_consensus.cc:2949] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.257213 20750 consensus_queue.cc:237] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:27.263859 20606 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:27.267211 21366 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:27.268401 21336 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:27.270579 21366 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:27.271879 21440 tablet_replica.cc:331] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:27.273357 21440 raft_consensus.cc:2238] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:27.274042 21440 raft_consensus.cc:2267] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:27.277387 21440 ts_tablet_manager.cc:1905] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:27.279806 21409 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.296726 21440 ts_tablet_manager.cc:1918] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:27.297250 21440 log.cc:1198] T 1d7658a3a42c44f485085e75a4bd7b88 P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/1d7658a3a42c44f485085e75a4bd7b88
I20250116 02:58:27.294481 20606 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.299023 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet 1d7658a3a42c44f485085e75a4bd7b88 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:27.300789 21336 raft_consensus.cc:2949] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.308964 20491 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 49f9e38171174b70b4373b56dbcd90be with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:27.313027 20506 catalog_manager.cc:5526] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:27.328683 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f9e38171174b70b4373b56dbcd90be with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:27.329763 21428 tablet_replica.cc:331] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:27.330983 21428 raft_consensus.cc:2238] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:27.331769 21428 raft_consensus.cc:2267] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:27.336400 21428 ts_tablet_manager.cc:1905] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:27.349622 21428 ts_tablet_manager.cc:1918] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:27.349958 21428 log.cc:1198] T 49f9e38171174b70b4373b56dbcd90be P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/49f9e38171174b70b4373b56dbcd90be
I20250116 02:58:27.351554 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet 49f9e38171174b70b4373b56dbcd90be (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:28.387018 20603 consensus_queue.cc:237] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:28.394181 20752 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:28.395294 20677 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:28.396193 20571 consensus_peers.cc:487] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8229db04eecb4bc3a9ff24fc8bcac934. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:28.396200 21215 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:28.397395 21443 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:28.405802 21412 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.408859 20677 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.407932 20752 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.420662 20682 consensus_queue.cc:237] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:28.427690 20505 catalog_manager.cc:5526] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.443260 20752 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:28.445027 21373 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:28.443385 20603 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:28.449558 20646 consensus_peers.cc:487] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe9318d6bdac457facfcaa97dea377f3. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:28.451035 21373 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:28.468086 21449 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.470204 20752 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.471431 20604 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.487973 20604 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:28.500474 20506 catalog_manager.cc:5526] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:28.540932 20571 consensus_peers.cc:487] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f69213cf66934f1a9aa19c9defd3a4d8. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:28.544377 20752 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:28.544674 20682 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:28.547046 21443 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:28.549201 21443 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:28.601279 21443 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.603726 20682 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.610333 20751 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.626868 20752 consensus_queue.cc:237] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:28.628449 20506 catalog_manager.cc:5526] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:28.635385 20603 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:28.635761 20720 consensus_peers.cc:487] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fd4be964f5040df93f3e609dddcc60a. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:28.636804 20682 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:28.637674 21455 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:28.639683 21441 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:28.655371 21409 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.657791 20604 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.660769 20682 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:28.682359 20503 catalog_manager.cc:5526] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:28.911491 21462 ts_tablet_manager.cc:927] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:28.914717 21462 tablet_copy_client.cc:323] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:28.916535 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet 8229db04eecb4bc3a9ff24fc8bcac934 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:28.917047 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet 8229db04eecb4bc3a9ff24fc8bcac934 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-8229db04eecb4bc3a9ff24fc8bcac934
I20250116 02:58:28.928220 20618 tablet_copy_source_session.cc:215] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:28.933568 21462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8229db04eecb4bc3a9ff24fc8bcac934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:28.949247 21462 tablet_copy_client.cc:806] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:28.949986 21462 tablet_copy_client.cc:670] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:28.954629 21462 tablet_copy_client.cc:538] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:28.965006 21462 tablet_bootstrap.cc:492] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:28.985015 21462 tablet_bootstrap.cc:492] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:28.985929 21462 tablet_bootstrap.cc:492] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:28.986663 21462 ts_tablet_manager.cc:1397] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.000s
I20250116 02:58:28.989395 21462 raft_consensus.cc:357] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:28.990412 21462 raft_consensus.cc:738] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:28.991226 21462 consensus_queue.cc:260] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:29.001643 21462 ts_tablet_manager.cc:1428] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20250116 02:58:29.003921 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-8229db04eecb4bc3a9ff24fc8bcac934 received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:29.003216 21467 ts_tablet_manager.cc:927] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:29.004431 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-8229db04eecb4bc3a9ff24fc8bcac934 on tablet 8229db04eecb4bc3a9ff24fc8bcac934 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:29.007652 21462 ts_tablet_manager.cc:927] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:29.012894 21467 tablet_copy_client.cc:323] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:29.013957 21462 tablet_copy_client.cc:323] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:29.014976 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet fe9318d6bdac457facfcaa97dea377f3 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:29.015619 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet fe9318d6bdac457facfcaa97dea377f3 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-fe9318d6bdac457facfcaa97dea377f3
I20250116 02:58:29.015720 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet f69213cf66934f1a9aa19c9defd3a4d8 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:29.016325 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet f69213cf66934f1a9aa19c9defd3a4d8 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-f69213cf66934f1a9aa19c9defd3a4d8
I20250116 02:58:29.023061 20618 tablet_copy_source_session.cc:215] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:29.023252 20693 tablet_copy_source_session.cc:215] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:29.029390 21462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69213cf66934f1a9aa19c9defd3a4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:29.031788 21467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9318d6bdac457facfcaa97dea377f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:29.046375 21462 tablet_copy_client.cc:806] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:29.047124 21462 tablet_copy_client.cc:670] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:29.054119 21462 tablet_copy_client.cc:538] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:29.063455 21462 tablet_bootstrap.cc:492] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:29.066180 21467 tablet_copy_client.cc:806] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:29.067067 21467 tablet_copy_client.cc:670] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:29.090055 21467 tablet_copy_client.cc:538] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:29.090438 21462 tablet_bootstrap.cc:492] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:29.104225 21462 tablet_bootstrap.cc:492] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:29.104954 21462 ts_tablet_manager.cc:1397] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.042s user 0.023s sys 0.004s
I20250116 02:58:29.109301 21462 raft_consensus.cc:357] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:29.110016 21462 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:29.106745 21467 tablet_bootstrap.cc:492] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:29.110566 21462 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:29.113384 21462 ts_tablet_manager.cc:1428] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250116 02:58:29.115458 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-f69213cf66934f1a9aa19c9defd3a4d8 received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:29.115926 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-f69213cf66934f1a9aa19c9defd3a4d8 on tablet f69213cf66934f1a9aa19c9defd3a4d8 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:29.135818 21467 tablet_bootstrap.cc:492] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:29.137151 21467 tablet_bootstrap.cc:492] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:29.137903 21467 ts_tablet_manager.cc:1397] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.009s
I20250116 02:58:29.141695 21467 raft_consensus.cc:357] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:29.142526 21467 raft_consensus.cc:738] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:29.143172 21467 consensus_queue.cc:260] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:29.145399 21467 ts_tablet_manager.cc:1428] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250116 02:58:29.200721 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-fe9318d6bdac457facfcaa97dea377f3 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:29.202183 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-fe9318d6bdac457facfcaa97dea377f3 on tablet fe9318d6bdac457facfcaa97dea377f3 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:29.220672 21462 ts_tablet_manager.cc:927] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:29.242435 21462 tablet_copy_client.cc:323] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.131:43647
I20250116 02:58:29.257089 20767 tablet_copy_service.cc:140] P 17446c8dacf04461acb843547665250f: Received BeginTabletCopySession request for tablet 3fd4be964f5040df93f3e609dddcc60a from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:35540)
I20250116 02:58:29.257931 20767 tablet_copy_service.cc:161] P 17446c8dacf04461acb843547665250f: Beginning new tablet copy session on tablet 3fd4be964f5040df93f3e609dddcc60a from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:35540: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-3fd4be964f5040df93f3e609dddcc60a
I20250116 02:58:29.273728 20767 tablet_copy_source_session.cc:215] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:29.279392 21462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd4be964f5040df93f3e609dddcc60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:29.307318 21462 tablet_copy_client.cc:806] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:29.309726 21462 tablet_copy_client.cc:670] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:29.317622 21462 tablet_copy_client.cc:538] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:29.329111 21462 tablet_bootstrap.cc:492] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:29.351783 21462 tablet_bootstrap.cc:492] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:29.352797 21462 tablet_bootstrap.cc:492] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:29.353551 21462 ts_tablet_manager.cc:1397] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.001s
I20250116 02:58:29.356773 21462 raft_consensus.cc:357] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:29.358551 21462 raft_consensus.cc:738] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:29.359357 21462 consensus_queue.cc:260] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:29.362483 21462 ts_tablet_manager.cc:1428] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250116 02:58:29.364428 20767 tablet_copy_service.cc:342] P 17446c8dacf04461acb843547665250f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-3fd4be964f5040df93f3e609dddcc60a received from {username='slave'} at 127.0.0.1:35540
I20250116 02:58:29.364899 20767 tablet_copy_service.cc:434] P 17446c8dacf04461acb843547665250f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-3fd4be964f5040df93f3e609dddcc60a on tablet 3fd4be964f5040df93f3e609dddcc60a with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:29.447863 21336 raft_consensus.cc:1212] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:29.531481 21336 raft_consensus.cc:1212] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:29.626020 21336 raft_consensus.cc:1212] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:29.737131 21336 raft_consensus.cc:1212] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:29.993727 21373 raft_consensus.cc:1059] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:29.996412 21373 consensus_queue.cc:237] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:30.017138 21336 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.018070 20606 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.021656 21443 raft_consensus.cc:1059] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:30.023176 21373 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250116 02:58:30.024335 21461 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:30.024689 21443 consensus_queue.cc:237] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:30.018682 20752 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.033840 21461 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:30.044394 21336 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.045531 20677 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.047804 20752 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.048552 21215 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:30.055115 21412 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250116 02:58:30.060082 21215 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:30.105577 21476 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.108949 21443 raft_consensus.cc:1059] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:30.108115 20750 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.112285 21443 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:30.122093 21474 raft_consensus.cc:1059] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
W20250116 02:58:30.132714 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 60006.2us
I20250116 02:58:30.130328 21474 consensus_queue.cc:237] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:30.142773 20605 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.172106 20503 catalog_manager.cc:5526] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.190196 21335 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.191257 20678 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.190073 21336 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.196802 21336 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.189663 21412 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.197983 20680 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.210165 21487 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:30.200261 21335 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.220088 20505 catalog_manager.cc:5526] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.221037 21481 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250116 02:58:30.233518 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 64143.3us
I20250116 02:58:30.236406 20605 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.239341 21474 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:30.244529 20681 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.264207 21474 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:30.275229 20682 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.279580 20750 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:30.272150 21491 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.286995 21474 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:30.287034 20749 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.292739 21215 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:30.310720 20753 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.317351 21455 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.320003 21336 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.319145 20682 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.337772 20605 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.402256 21335 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.404841 20681 consensus_queue.cc:237] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:30.414343 21334 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:30.417706 21476 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:30.420112 20753 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:30.404891 20503 catalog_manager.cc:5526] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:30.436358 21476 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:30.433969 21461 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.442328 21334 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.458767 20505 catalog_manager.cc:5526] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:30.460924 20749 raft_consensus.cc:2949] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.467702 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet fe9318d6bdac457facfcaa97dea377f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:30.469990 20606 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:30.475554 20605 consensus_queue.cc:237] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:30.482650 21334 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:30.482849 20682 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:30.485538 21481 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:30.486021 20680 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:30.487577 21481 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:30.491070 21215 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:30.492712 20753 consensus_queue.cc:237] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:30.501255 21335 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:30.498102 20503 catalog_manager.cc:5526] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.510387 21455 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:30.509941 21491 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.513744 20680 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:30.520340 21474 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:30.521893 21334 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:30.506965 21412 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.523427 21490 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.537145 20491 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 3fd4be964f5040df93f3e609dddcc60a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:30.539065 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet fe9318d6bdac457facfcaa97dea377f3 with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:30.539973 21491 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:30.542017 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f69213cf66934f1a9aa19c9defd3a4d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:30.548053 21336 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.564309 21503 tablet_replica.cc:331] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:30.566110 21503 raft_consensus.cc:2238] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:30.567376 21503 raft_consensus.cc:2267] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:30.571193 21335 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.573376 20680 raft_consensus.cc:2949] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.577849 21503 ts_tablet_manager.cc:1905] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:30.545153 21334 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.583282 20682 raft_consensus.cc:2949] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.591444 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8229db04eecb4bc3a9ff24fc8bcac934 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:30.597929 20504 catalog_manager.cc:5526] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:30.606873 20505 catalog_manager.cc:5526] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.627396 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet f69213cf66934f1a9aa19c9defd3a4d8 with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:30.643958 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd4be964f5040df93f3e609dddcc60a with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:30.647670 20506 catalog_manager.cc:5526] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:30.650880 21505 tablet_replica.cc:331] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:30.651830 21505 raft_consensus.cc:2238] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:30.649187 21503 ts_tablet_manager.cc:1918] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:30.652407 21505 raft_consensus.cc:2267] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:30.654378 20678 raft_consensus.cc:2949] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:30.655557 21503 log.cc:1198] T fe9318d6bdac457facfcaa97dea377f3 P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/fe9318d6bdac457facfcaa97dea377f3
I20250116 02:58:30.657420 21503 tablet_replica.cc:331] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:30.658958 21503 raft_consensus.cc:2238] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:30.657318 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet fe9318d6bdac457facfcaa97dea377f3 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:30.659807 21503 raft_consensus.cc:2267] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:30.664052 21503 ts_tablet_manager.cc:1905] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:30.666518 21505 ts_tablet_manager.cc:1905] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:30.672195 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet 8229db04eecb4bc3a9ff24fc8bcac934 with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:30.687266 21503 ts_tablet_manager.cc:1918] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:30.687980 21503 log.cc:1198] T 3fd4be964f5040df93f3e609dddcc60a P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/3fd4be964f5040df93f3e609dddcc60a
I20250116 02:58:30.689898 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet 3fd4be964f5040df93f3e609dddcc60a (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:30.702018 21505 ts_tablet_manager.cc:1918] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:30.702523 21505 log.cc:1198] T f69213cf66934f1a9aa19c9defd3a4d8 P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/f69213cf66934f1a9aa19c9defd3a4d8
I20250116 02:58:30.704574 21505 tablet_replica.cc:331] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:30.704630 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet f69213cf66934f1a9aa19c9defd3a4d8 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:30.705919 21505 raft_consensus.cc:2238] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:30.706689 21505 raft_consensus.cc:2267] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:30.711668 21505 ts_tablet_manager.cc:1905] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:30.726089 21505 ts_tablet_manager.cc:1918] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:30.726496 21505 log.cc:1198] T 8229db04eecb4bc3a9ff24fc8bcac934 P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/8229db04eecb4bc3a9ff24fc8bcac934
I20250116 02:58:30.727903 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet 8229db04eecb4bc3a9ff24fc8bcac934 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
W20250116 02:58:31.556974 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 46923.7us
I20250116 02:58:31.696364 20753 consensus_queue.cc:237] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:31.702093 20607 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:31.702406 20682 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:31.702890 20720 consensus_peers.cc:487] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ef2f3fbb2a04b7fb68c0bd275697181. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:31.704242 21513 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:31.705471 21441 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:31.715862 21455 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.723539 20680 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.723589 20607 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.732338 20682 consensus_queue.cc:237] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:31.753805 20753 raft_consensus.cc:1270] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:31.754212 20646 consensus_peers.cc:487] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54d83a73e060487a8cb3bf72207828c2. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:31.748513 20505 catalog_manager.cc:5526] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:31.755632 21515 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:31.765332 20607 raft_consensus.cc:1270] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:31.767724 21373 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:31.769702 21507 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.772140 20750 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.796311 20750 consensus_queue.cc:237] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:31.797886 20604 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:31.810107 20720 consensus_peers.cc:487] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d25e9e5e0954623a098bf759e01120b. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:31.812354 20682 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:31.816254 21519 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:31.818912 20604 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:31.823028 21519 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:31.822971 20506 catalog_manager.cc:5526] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.838773 21519 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.845036 20682 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.855194 20604 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.874320 20506 catalog_manager.cc:5526] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:31.859189 20607 consensus_queue.cc:237] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:31.907931 20753 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:31.908267 20571 consensus_peers.cc:487] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b57de7b372d4101b7f88debce8eea57. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:31.916363 21481 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:31.918394 20682 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:31.920290 21481 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:31.937080 21520 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.939008 20682 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.940788 20753 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:31.960680 20505 catalog_manager.cc:5526] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:32.173511 20704 debug-util.cc:398] Leaking SignalData structure 0x7b08000fc840 after lost signal to thread 19389
W20250116 02:58:32.174593 20704 debug-util.cc:398] Leaking SignalData structure 0x7b08001688e0 after lost signal to thread 20540
W20250116 02:58:32.175853 20704 debug-util.cc:398] Leaking SignalData structure 0x7b0800306260 after lost signal to thread 20632
W20250116 02:58:32.176851 20704 debug-util.cc:398] Leaking SignalData structure 0x7b0800309220 after lost signal to thread 20707
W20250116 02:58:32.177768 20704 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff180 after lost signal to thread 20781
W20250116 02:58:32.179049 20704 debug-util.cc:398] Leaking SignalData structure 0x7b080030f7e0 after lost signal to thread 21360
I20250116 02:58:32.239544 21538 ts_tablet_manager.cc:927] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:32.244242 21538 tablet_copy_client.cc:323] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:32.246488 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 54d83a73e060487a8cb3bf72207828c2 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:32.247121 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 54d83a73e060487a8cb3bf72207828c2 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-54d83a73e060487a8cb3bf72207828c2
I20250116 02:58:32.263404 20693 tablet_copy_source_session.cc:215] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:32.265214 21540 ts_tablet_manager.cc:927] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:32.268790 21538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d83a73e060487a8cb3bf72207828c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:32.295337 21540 tablet_copy_client.cc:323] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.131:43647
I20250116 02:58:32.303200 20767 tablet_copy_service.cc:140] P 17446c8dacf04461acb843547665250f: Received BeginTabletCopySession request for tablet 6ef2f3fbb2a04b7fb68c0bd275697181 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:35540)
I20250116 02:58:32.303992 21542 ts_tablet_manager.cc:927] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:32.304188 20767 tablet_copy_service.cc:161] P 17446c8dacf04461acb843547665250f: Beginning new tablet copy session on tablet 6ef2f3fbb2a04b7fb68c0bd275697181 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:35540: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-6ef2f3fbb2a04b7fb68c0bd275697181
I20250116 02:58:32.312959 21542 tablet_copy_client.cc:323] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.131:43647
I20250116 02:58:32.315095 21538 tablet_copy_client.cc:806] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:32.315048 20766 tablet_copy_service.cc:140] P 17446c8dacf04461acb843547665250f: Received BeginTabletCopySession request for tablet 2d25e9e5e0954623a098bf759e01120b from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:35540)
I20250116 02:58:32.316089 21538 tablet_copy_client.cc:670] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:32.316064 20766 tablet_copy_service.cc:161] P 17446c8dacf04461acb843547665250f: Beginning new tablet copy session on tablet 2d25e9e5e0954623a098bf759e01120b from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:35540: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-2d25e9e5e0954623a098bf759e01120b
I20250116 02:58:32.319907 20767 tablet_copy_source_session.cc:215] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:32.323722 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef2f3fbb2a04b7fb68c0bd275697181. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:32.344946 21540 tablet_copy_client.cc:806] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:32.345764 21540 tablet_copy_client.cc:670] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:32.350116 21540 tablet_copy_client.cc:538] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:32.351233 20766 tablet_copy_source_session.cc:215] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:32.362766 21538 tablet_copy_client.cc:538] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:32.365437 21540 tablet_bootstrap.cc:492] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:32.386049 21542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d25e9e5e0954623a098bf759e01120b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250116 02:58:32.394460 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 49294.6us
I20250116 02:58:32.423619 21538 tablet_bootstrap.cc:492] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:32.430569 21542 tablet_copy_client.cc:806] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:32.433244 21542 tablet_copy_client.cc:670] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:32.451174 21542 tablet_copy_client.cc:538] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:32.455541 21540 tablet_bootstrap.cc:492] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:32.458541 21540 tablet_bootstrap.cc:492] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:32.459662 21540 ts_tablet_manager.cc:1397] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.097s user 0.028s sys 0.013s
I20250116 02:58:32.463253 21542 tablet_bootstrap.cc:492] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:32.463680 21540 raft_consensus.cc:357] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:32.464883 21540 raft_consensus.cc:738] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:32.465660 21540 consensus_queue.cc:260] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:32.470566 21538 tablet_bootstrap.cc:492] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:32.471684 21538 tablet_bootstrap.cc:492] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:32.483387 21538 ts_tablet_manager.cc:1397] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.060s user 0.019s sys 0.008s
I20250116 02:58:32.486824 21540 ts_tablet_manager.cc:1428] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.027s user 0.006s sys 0.005s
I20250116 02:58:32.486719 21538 raft_consensus.cc:357] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:32.493419 21538 raft_consensus.cc:738] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:32.494217 21538 consensus_queue.cc:260] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:32.501688 20766 tablet_copy_service.cc:342] P 17446c8dacf04461acb843547665250f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-6ef2f3fbb2a04b7fb68c0bd275697181 received from {username='slave'} at 127.0.0.1:35540
I20250116 02:58:32.502192 20766 tablet_copy_service.cc:434] P 17446c8dacf04461acb843547665250f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-6ef2f3fbb2a04b7fb68c0bd275697181 on tablet 6ef2f3fbb2a04b7fb68c0bd275697181 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:32.503901 21542 tablet_bootstrap.cc:492] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:32.504937 21542 tablet_bootstrap.cc:492] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:32.505664 21542 ts_tablet_manager.cc:1397] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.043s user 0.022s sys 0.009s
I20250116 02:58:32.506088 21540 ts_tablet_manager.cc:927] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:32.509660 21542 raft_consensus.cc:357] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:32.510816 21542 raft_consensus.cc:738] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:32.513275 21542 consensus_queue.cc:260] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:32.516884 21542 ts_tablet_manager.cc:1428] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250116 02:58:32.518747 20766 tablet_copy_service.cc:342] P 17446c8dacf04461acb843547665250f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-2d25e9e5e0954623a098bf759e01120b received from {username='slave'} at 127.0.0.1:35540
I20250116 02:58:32.519172 20766 tablet_copy_service.cc:434] P 17446c8dacf04461acb843547665250f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-2d25e9e5e0954623a098bf759e01120b on tablet 2d25e9e5e0954623a098bf759e01120b with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:32.529292 21538 ts_tablet_manager.cc:1428] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.045s user 0.017s sys 0.005s
I20250116 02:58:32.531945 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-54d83a73e060487a8cb3bf72207828c2 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:32.532469 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-54d83a73e060487a8cb3bf72207828c2 on tablet 54d83a73e060487a8cb3bf72207828c2 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:32.535228 21540 tablet_copy_client.cc:323] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:32.537191 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet 0b57de7b372d4101b7f88debce8eea57 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:32.537729 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet 0b57de7b372d4101b7f88debce8eea57 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-0b57de7b372d4101b7f88debce8eea57
I20250116 02:58:32.548898 20618 tablet_copy_source_session.cc:215] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:32.552680 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b57de7b372d4101b7f88debce8eea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:32.570192 21540 tablet_copy_client.cc:806] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:32.570811 21540 tablet_copy_client.cc:670] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:32.574899 21540 tablet_copy_client.cc:538] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:32.593431 21540 tablet_bootstrap.cc:492] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:32.615641 21540 tablet_bootstrap.cc:492] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:32.616737 21540 tablet_bootstrap.cc:492] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:32.617532 21540 ts_tablet_manager.cc:1397] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.004s
I20250116 02:58:32.620319 21540 raft_consensus.cc:357] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:32.621207 21540 raft_consensus.cc:738] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:32.621958 21540 consensus_queue.cc:260] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:32.625102 21540 ts_tablet_manager.cc:1428] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20250116 02:58:32.629799 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-0b57de7b372d4101b7f88debce8eea57 received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:32.630308 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-0b57de7b372d4101b7f88debce8eea57 on tablet 0b57de7b372d4101b7f88debce8eea57 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:32.811955 21336 raft_consensus.cc:1212] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:32.837524 21336 raft_consensus.cc:1212] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:32.847502 21336 raft_consensus.cc:1212] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:32.921468 21336 raft_consensus.cc:1212] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:33.318451 21506 raft_consensus.cc:1059] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:33.320757 21506 consensus_queue.cc:237] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:33.339239 21336 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.340979 20749 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.341925 21481 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:33.344040 21481 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:33.358559 21553 raft_consensus.cc:1059] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:33.364485 21553 consensus_queue.cc:237] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:33.368989 20682 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.371090 21520 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:33.373772 21520 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.376065 21334 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.393659 20750 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.401013 20505 catalog_manager.cc:5526] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:33.415869 21511 raft_consensus.cc:1059] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:33.431932 21511 consensus_queue.cc:237] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:33.438156 21559 raft_consensus.cc:1059] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:33.441330 20682 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.441187 21559 consensus_queue.cc:237] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:33.461064 21336 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.465921 20677 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.468540 21559 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:33.469859 20608 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.470382 21559 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:33.475289 21513 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:33.548298 20607 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.550264 21559 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:33.568807 21336 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.567948 21519 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.581929 20680 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.625038 21336 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.637029 21334 raft_consensus.cc:1270] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.638876 20607 raft_consensus.cc:1270] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.644631 21568 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:33.648075 21570 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:33.661001 20608 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.664542 20680 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.701121 20753 raft_consensus.cc:1270] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:33.701121 21519 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20250116 02:58:33.723031 21553 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:33.736510 21572 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.753047 21515 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:33.745383 21336 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:33.808313 20749 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 02:58:33.905119 21567 log.cc:926] Time spent T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.095s user 0.000s sys 0.003s
I20250116 02:58:34.066808 21581 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.087646 20505 catalog_manager.cc:5526] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.089941 20505 catalog_manager.cc:5526] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.107646 21334 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.116617 20676 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.155855 20607 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 02:58:34.183845 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 73289.8us
I20250116 02:58:34.188091 20504 catalog_manager.cc:5526] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.199604 20605 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.214736 20603 consensus_queue.cc:237] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:34.223989 20677 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:34.227954 21506 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:34.242941 20749 consensus_queue.cc:237] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:34.244107 21521 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.245682 20677 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.256361 20752 consensus_queue.cc:237] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:34.257023 21334 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:34.258500 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 0b57de7b372d4101b7f88debce8eea57 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:34.268816 21589 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:34.274374 21335 raft_consensus.cc:2949] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.281674 20606 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:34.263460 21336 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:34.276798 21333 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:34.283334 21559 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:34.294322 20601 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:34.297737 21590 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:34.294540 20502 catalog_manager.cc:5526] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.308622 21590 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.311908 20607 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.316274 21582 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.328014 20605 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.334410 21581 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:34.338023 21519 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:34.361012 21333 raft_consensus.cc:2949] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.363255 21335 raft_consensus.cc:2949] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.365799 20491 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 6ef2f3fbb2a04b7fb68c0bd275697181 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:34.386997 20491 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 2d25e9e5e0954623a098bf759e01120b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:34.400211 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b57de7b372d4101b7f88debce8eea57 with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:34.414666 20502 catalog_manager.cc:5526] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:34.417341 20502 catalog_manager.cc:5526] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:34.444136 20677 consensus_queue.cc:237] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:34.440506 21593 tablet_replica.cc:331] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:34.450057 21335 raft_consensus.cc:1270] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:34.450184 21593 raft_consensus.cc:2238] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:34.472548 21593 raft_consensus.cc:2267] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:34.473054 20749 raft_consensus.cc:1270] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:34.482249 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef2f3fbb2a04b7fb68c0bd275697181 with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:34.476120 21593 ts_tablet_manager.cc:1905] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:34.491899 21594 tablet_replica.cc:331] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:34.472730 21592 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20250116 02:58:34.495050 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d25e9e5e0954623a098bf759e01120b with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:34.517860 21593 ts_tablet_manager.cc:1918] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:34.522289 21572 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:34.518489 21593 log.cc:1198] T 0b57de7b372d4101b7f88debce8eea57 P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/0b57de7b372d4101b7f88debce8eea57
I20250116 02:58:34.528800 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet 0b57de7b372d4101b7f88debce8eea57 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:34.531277 21594 raft_consensus.cc:2238] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:34.532271 21594 raft_consensus.cc:2267] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:34.536247 21594 ts_tablet_manager.cc:1905] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:34.536979 21572 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.542001 21335 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.547371 20752 raft_consensus.cc:2949] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.559263 20502 catalog_manager.cc:5526] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:34.586828 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d83a73e060487a8cb3bf72207828c2 with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:34.641393 21595 tablet_replica.cc:331] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:34.646755 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 54d83a73e060487a8cb3bf72207828c2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:34.670919 21595 raft_consensus.cc:2238] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:34.671885 21595 raft_consensus.cc:2267] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:34.676807 21595 ts_tablet_manager.cc:1905] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:34.683910 21594 ts_tablet_manager.cc:1918] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:34.704085 21594 log.cc:1198] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/6ef2f3fbb2a04b7fb68c0bd275697181
I20250116 02:58:34.707840 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet 6ef2f3fbb2a04b7fb68c0bd275697181 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:34.707742 21594 tablet_replica.cc:331] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:34.711659 21594 raft_consensus.cc:2238] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:34.712549 21594 raft_consensus.cc:2267] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:34.717099 21594 ts_tablet_manager.cc:1905] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:34.739624 21595 ts_tablet_manager.cc:1918] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:34.742939 21595 log.cc:1198] T 54d83a73e060487a8cb3bf72207828c2 P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/54d83a73e060487a8cb3bf72207828c2
I20250116 02:58:34.745227 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet 54d83a73e060487a8cb3bf72207828c2 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:34.740919 21594 ts_tablet_manager.cc:1918] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:34.752518 21594 log.cc:1198] T 2d25e9e5e0954623a098bf759e01120b P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/2d25e9e5e0954623a098bf759e01120b
I20250116 02:58:34.754242 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet 2d25e9e5e0954623a098bf759e01120b (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:35.753978 20753 consensus_queue.cc:237] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:35.770042 20676 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:35.773357 20720 consensus_peers.cc:487] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb4a7093fd744af792f09b82b0946a8e. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:35.773830 21597 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:35.789144 20605 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:35.790953 21581 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:35.793466 21590 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:35.800783 20676 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:35.829448 20677 consensus_queue.cc:237] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:35.840356 20607 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:35.850230 20505 catalog_manager.cc:5526] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:35.857733 20646 consensus_peers.cc:487] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 852e609a81a14d53bed4660b6e8b47f7. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:35.866195 20749 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:35.866570 20607 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:35.867903 21599 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:35.869766 21572 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:35.891368 21599 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:35.902827 20749 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:35.910893 20607 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:35.930804 20676 consensus_queue.cc:237] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
W20250116 02:58:35.943131 20646 consensus_peers.cc:487] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 461ef3aac67e4aa49204ce07ff551513. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:35.946060 20607 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:35.947690 20749 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:35.948109 21568 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:35.950421 21568 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:35.968865 20502 catalog_manager.cc:5526] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:36.022802 21599 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:36.027947 20605 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:36.058936 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 92815.3us
I20250116 02:58:36.067876 20502 catalog_manager.cc:5526] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:36.073218 20752 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:36.071734 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 59640.5us
I20250116 02:58:36.088114 20605 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:36.108116 20749 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:36.111953 21611 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250116 02:58:36.116763 20571 consensus_peers.cc:487] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03c04d7691954d8e9278d9ba71f359fe. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:36.122063 20676 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:36.124039 21615 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:36.131821 21616 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:36.133349 20749 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:36.155155 20681 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:36.205071 20505 catalog_manager.cc:5526] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:36.232332 21621 ts_tablet_manager.cc:927] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:36.237802 21621 tablet_copy_client.cc:323] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.131:43647
I20250116 02:58:36.239650 20766 tablet_copy_service.cc:140] P 17446c8dacf04461acb843547665250f: Received BeginTabletCopySession request for tablet eb4a7093fd744af792f09b82b0946a8e from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:35540)
I20250116 02:58:36.240257 20766 tablet_copy_service.cc:161] P 17446c8dacf04461acb843547665250f: Beginning new tablet copy session on tablet eb4a7093fd744af792f09b82b0946a8e from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:35540: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-eb4a7093fd744af792f09b82b0946a8e
I20250116 02:58:36.254118 20766 tablet_copy_source_session.cc:215] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:36.257512 21621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb4a7093fd744af792f09b82b0946a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:36.281117 21621 tablet_copy_client.cc:806] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:36.281836 21621 tablet_copy_client.cc:670] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:36.286057 21621 tablet_copy_client.cc:538] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:36.304140 21621 tablet_bootstrap.cc:492] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:36.351796 21625 ts_tablet_manager.cc:927] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:36.395987 21623 ts_tablet_manager.cc:927] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:36.442255 21625 tablet_copy_client.cc:323] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:36.444556 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 852e609a81a14d53bed4660b6e8b47f7 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:36.445266 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 852e609a81a14d53bed4660b6e8b47f7 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-852e609a81a14d53bed4660b6e8b47f7
I20250116 02:58:36.467296 21623 tablet_copy_client.cc:323] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:36.469426 20692 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 461ef3aac67e4aa49204ce07ff551513 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:36.470041 20692 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 461ef3aac67e4aa49204ce07ff551513 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-461ef3aac67e4aa49204ce07ff551513
I20250116 02:58:36.475989 20693 tablet_copy_source_session.cc:215] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:36.492022 21621 tablet_bootstrap.cc:492] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:36.492887 21625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 852e609a81a14d53bed4660b6e8b47f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:36.493114 21621 tablet_bootstrap.cc:492] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:36.495402 21621 ts_tablet_manager.cc:1397] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.192s user 0.077s sys 0.108s
I20250116 02:58:36.498719 21621 raft_consensus.cc:357] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:36.499706 21621 raft_consensus.cc:738] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:36.500399 21621 consensus_queue.cc:260] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:36.503014 20692 tablet_copy_source_session.cc:215] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:36.510334 21623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 461ef3aac67e4aa49204ce07ff551513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:36.569521 21629 ts_tablet_manager.cc:927] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:36.581017 21621 ts_tablet_manager.cc:1428] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.085s user 0.022s sys 0.051s
I20250116 02:58:36.588470 20766 tablet_copy_service.cc:342] P 17446c8dacf04461acb843547665250f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-eb4a7093fd744af792f09b82b0946a8e received from {username='slave'} at 127.0.0.1:35540
I20250116 02:58:36.589025 20766 tablet_copy_service.cc:434] P 17446c8dacf04461acb843547665250f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-eb4a7093fd744af792f09b82b0946a8e on tablet eb4a7093fd744af792f09b82b0946a8e with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:36.595747 21625 tablet_copy_client.cc:806] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:36.596498 21625 tablet_copy_client.cc:670] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:36.597991 21629 tablet_copy_client.cc:323] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:36.602421 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet 03c04d7691954d8e9278d9ba71f359fe from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:36.603096 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet 03c04d7691954d8e9278d9ba71f359fe from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-03c04d7691954d8e9278d9ba71f359fe
I20250116 02:58:36.615756 21623 tablet_copy_client.cc:806] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:36.616547 21623 tablet_copy_client.cc:670] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:36.624634 21623 tablet_copy_client.cc:538] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:36.613837 21625 tablet_copy_client.cc:538] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:36.637905 20618 tablet_copy_source_session.cc:215] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:36.644301 21629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c04d7691954d8e9278d9ba71f359fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:36.650466 21623 tablet_bootstrap.cc:492] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:36.657495 21333 raft_consensus.cc:1212] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:36.658512 21625 tablet_bootstrap.cc:492] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:36.702267 21629 tablet_copy_client.cc:806] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:36.705215 21625 tablet_bootstrap.cc:492] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:36.703006 21629 tablet_copy_client.cc:670] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:36.714869 21625 tablet_bootstrap.cc:492] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:36.715734 21625 ts_tablet_manager.cc:1397] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.057s user 0.018s sys 0.014s
I20250116 02:58:36.720422 21629 tablet_copy_client.cc:538] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:36.731804 21625 raft_consensus.cc:357] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:36.733428 21625 raft_consensus.cc:738] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:36.734242 21625 consensus_queue.cc:260] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:36.733043 21629 tablet_bootstrap.cc:492] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:36.740998 21625 ts_tablet_manager.cc:1428] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20250116 02:58:36.752034 20692 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-852e609a81a14d53bed4660b6e8b47f7 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:36.752584 20692 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-852e609a81a14d53bed4660b6e8b47f7 on tablet 852e609a81a14d53bed4660b6e8b47f7 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:36.774583 21629 tablet_bootstrap.cc:492] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:36.775925 21629 tablet_bootstrap.cc:492] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:36.776813 21629 ts_tablet_manager.cc:1397] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.011s
I20250116 02:58:36.778532 21623 tablet_bootstrap.cc:492] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:36.780107 21629 raft_consensus.cc:357] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:36.781248 21629 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:36.782155 21629 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:36.786298 21623 tablet_bootstrap.cc:492] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:36.787326 21623 ts_tablet_manager.cc:1397] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.138s user 0.010s sys 0.015s
I20250116 02:58:36.788157 21629 ts_tablet_manager.cc:1428] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250116 02:58:36.792496 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-03c04d7691954d8e9278d9ba71f359fe received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:36.792438 21623 raft_consensus.cc:357] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:36.793201 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-03c04d7691954d8e9278d9ba71f359fe on tablet 03c04d7691954d8e9278d9ba71f359fe with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:36.794402 21623 raft_consensus.cc:738] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:36.795224 21623 consensus_queue.cc:260] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:36.797900 21623 ts_tablet_manager.cc:1428] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250116 02:58:36.799988 20692 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-461ef3aac67e4aa49204ce07ff551513 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:36.800453 20692 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-461ef3aac67e4aa49204ce07ff551513 on tablet 461ef3aac67e4aa49204ce07ff551513 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:36.954555 21333 raft_consensus.cc:1212] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:36.955164 21336 raft_consensus.cc:1212] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:36.982918 21333 raft_consensus.cc:1212] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:37.169595 21631 raft_consensus.cc:1059] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:37.172628 21631 consensus_queue.cc:237] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:37.181331 21333 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.182049 20677 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.184131 21581 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:37.185065 21631 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:37.190188 20607 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.221027 21631 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:37.239151 21581 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.241139 20605 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.250644 20677 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.258913 21333 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.265384 20502 catalog_manager.cc:5526] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.294704 20749 consensus_queue.cc:237] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:37.309489 21333 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:37.311903 21581 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:37.320581 21631 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.328361 20603 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:37.329780 21336 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.337028 21581 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:37.341162 20491 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet eb4a7093fd744af792f09b82b0946a8e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:37.343613 20605 raft_consensus.cc:2949] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.349282 20504 catalog_manager.cc:5526] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.361748 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4a7093fd744af792f09b82b0946a8e with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:37.369387 21599 raft_consensus.cc:1059] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:37.372332 21599 consensus_queue.cc:237] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:37.382117 21646 tablet_replica.cc:331] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:37.386463 21336 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.390545 21568 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:37.394402 20749 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.395864 20603 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.396099 21613 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:37.400064 21646 raft_consensus.cc:2238] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:37.403208 21646 raft_consensus.cc:2267] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:37.403095 21613 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:37.407455 21646 ts_tablet_manager.cc:1905] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:37.409044 21373 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.436440 21568 raft_consensus.cc:1059] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:37.438002 21646 ts_tablet_manager.cc:1918] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:37.438652 21646 log.cc:1198] T eb4a7093fd744af792f09b82b0946a8e P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/eb4a7093fd744af792f09b82b0946a8e
I20250116 02:58:37.433034 20749 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.442366 21568 consensus_queue.cc:237] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:37.442332 20603 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.447053 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet eb4a7093fd744af792f09b82b0946a8e (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:37.448419 21336 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.452630 20752 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.454233 21333 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.455973 21651 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:37.462396 20504 catalog_manager.cc:5526] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.470638 20603 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.467698 21599 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20250116 02:58:37.474941 21613 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:37.517017 20677 consensus_queue.cc:237] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:37.540735 21616 raft_consensus.cc:1059] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:37.543493 21616 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:37.547680 21651 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.554294 20749 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.561268 20751 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.560456 21336 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.563290 21637 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:37.567790 21620 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:37.573645 20605 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.581748 20504 catalog_manager.cc:5526] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:37.607586 21336 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:37.609265 21651 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:37.614682 20751 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:37.616917 21613 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.618256 21599 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:37.624361 20751 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.626824 20677 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:37.640090 21616 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:37.642395 21336 raft_consensus.cc:2949] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.650280 21657 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.657980 21333 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.659442 20753 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.662827 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 852e609a81a14d53bed4660b6e8b47f7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:37.677642 21334 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.685922 20503 catalog_manager.cc:5526] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.697556 20505 catalog_manager.cc:5526] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.694916 20676 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.715312 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet 852e609a81a14d53bed4660b6e8b47f7 with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:37.750285 20681 consensus_queue.cc:237] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:37.772202 21334 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:37.775318 21599 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:37.776427 20603 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:37.778371 21599 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:37.791179 21651 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.793650 20606 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.803473 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 461ef3aac67e4aa49204ce07ff551513 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:37.813401 20504 catalog_manager.cc:5526] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:37.814721 21333 raft_consensus.cc:2949] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.817807 21662 tablet_replica.cc:331] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:37.819058 21662 raft_consensus.cc:2238] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:37.819794 21662 raft_consensus.cc:2267] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:37.823949 21662 ts_tablet_manager.cc:1905] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:37.826402 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet 461ef3aac67e4aa49204ce07ff551513 with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:37.828588 21663 tablet_replica.cc:331] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:37.831337 21663 raft_consensus.cc:2238] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:37.834214 21663 raft_consensus.cc:2267] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:37.852872 21663 ts_tablet_manager.cc:1905] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:37.861208 20606 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:37.873214 21662 ts_tablet_manager.cc:1918] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:37.873864 21662 log.cc:1198] T 852e609a81a14d53bed4660b6e8b47f7 P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/852e609a81a14d53bed4660b6e8b47f7
I20250116 02:58:37.876010 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet 852e609a81a14d53bed4660b6e8b47f7 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:37.879897 21663 ts_tablet_manager.cc:1918] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:37.880569 21663 log.cc:1198] T 461ef3aac67e4aa49204ce07ff551513 P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/461ef3aac67e4aa49204ce07ff551513
I20250116 02:58:37.884737 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet 461ef3aac67e4aa49204ce07ff551513 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:37.885854 21333 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:37.886715 20681 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:37.887492 21658 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:37.888635 21637 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:37.896505 21616 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.898291 20681 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.898660 21333 raft_consensus.cc:2949] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.911073 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 03c04d7691954d8e9278d9ba71f359fe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:37.916712 20505 catalog_manager.cc:5526] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:37.934633 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c04d7691954d8e9278d9ba71f359fe with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:37.935674 21663 tablet_replica.cc:331] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:37.936637 21663 raft_consensus.cc:2238] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:37.937312 21663 raft_consensus.cc:2267] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:37.942056 21663 ts_tablet_manager.cc:1905] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:37.968487 21663 ts_tablet_manager.cc:1918] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:37.969031 21663 log.cc:1198] T 03c04d7691954d8e9278d9ba71f359fe P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/03c04d7691954d8e9278d9ba71f359fe
I20250116 02:58:37.970686 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet 03c04d7691954d8e9278d9ba71f359fe (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:38.988204 20753 consensus_queue.cc:237] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
W20250116 02:58:38.995218 20720 consensus_peers.cc:487] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99adad9dddaa4121a293aef6fccd1694. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:38.996048 20681 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:38.996524 20606 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:38.997963 21631 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:38.999001 21666 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:39.012727 21666 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.016762 20681 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.025058 20606 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.037293 20677 consensus_queue.cc:237] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.034471 20505 catalog_manager.cc:5526] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:39.052575 20646 consensus_peers.cc:487] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da8115c09ba24b95a9969edcc6ca4d8c. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:39.056201 20605 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:39.057866 21651 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:39.059613 20753 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:39.063683 21651 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:39.072168 21651 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.075749 20605 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.101672 20677 consensus_queue.cc:237] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.111830 20753 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.119714 20504 catalog_manager.cc:5526] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:39.141057 20646 consensus_peers.cc:487] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19c9bd4e72374377958093c605e5c6ae. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:39.141419 20753 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:39.141480 20605 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:39.143146 21676 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:39.144258 21664 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:39.158707 20606 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.157256 21676 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.165096 20753 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.176420 20605 consensus_queue.cc:237] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.181368 20503 catalog_manager.cc:5526] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.196687 20749 raft_consensus.cc:1270] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:39.202014 21658 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250116 02:58:39.204082 20571 consensus_peers.cc:487] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d8e1b8edadd42f08808ffaf11b40c8e. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:39.205689 20676 raft_consensus.cc:1270] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:39.213733 21682 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:39.217008 21616 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.218210 20753 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.241165 20503 catalog_manager.cc:5526] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:39.284219 20676 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:39.482872 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 44821.7us
I20250116 02:58:39.513477 21689 ts_tablet_manager.cc:927] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:39.519609 21689 tablet_copy_client.cc:323] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:39.521613 20692 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet da8115c09ba24b95a9969edcc6ca4d8c from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:39.522286 20692 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet da8115c09ba24b95a9969edcc6ca4d8c from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-da8115c09ba24b95a9969edcc6ca4d8c
I20250116 02:58:39.537319 20692 tablet_copy_source_session.cc:215] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:39.543059 21689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8115c09ba24b95a9969edcc6ca4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:39.574513 21689 tablet_copy_client.cc:806] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:39.582333 21689 tablet_copy_client.cc:670] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:39.587539 21689 tablet_copy_client.cc:538] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:39.600387 21689 tablet_bootstrap.cc:492] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:39.631827 21691 ts_tablet_manager.cc:927] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:39.641594 21691 tablet_copy_client.cc:323] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.131:43647
I20250116 02:58:39.643641 20766 tablet_copy_service.cc:140] P 17446c8dacf04461acb843547665250f: Received BeginTabletCopySession request for tablet 99adad9dddaa4121a293aef6fccd1694 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:35540)
I20250116 02:58:39.644286 20766 tablet_copy_service.cc:161] P 17446c8dacf04461acb843547665250f: Beginning new tablet copy session on tablet 99adad9dddaa4121a293aef6fccd1694 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:35540: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-99adad9dddaa4121a293aef6fccd1694
I20250116 02:58:39.646395 21689 tablet_bootstrap.cc:492] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:39.649572 21689 tablet_bootstrap.cc:492] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:39.650632 21689 ts_tablet_manager.cc:1397] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.051s user 0.024s sys 0.002s
I20250116 02:58:39.655084 20766 tablet_copy_source_session.cc:215] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:39.654361 21689 raft_consensus.cc:357] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.655746 21689 raft_consensus.cc:738] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:39.656426 21689 consensus_queue.cc:260] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.659220 21691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99adad9dddaa4121a293aef6fccd1694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:39.666509 21689 ts_tablet_manager.cc:1428] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20250116 02:58:39.671756 20692 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-da8115c09ba24b95a9969edcc6ca4d8c received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:39.672322 20692 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-da8115c09ba24b95a9969edcc6ca4d8c on tablet da8115c09ba24b95a9969edcc6ca4d8c with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:39.693121 21691 tablet_copy_client.cc:806] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:39.694075 21691 tablet_copy_client.cc:670] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:39.699276 21689 ts_tablet_manager.cc:927] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:39.701936 21691 tablet_copy_client.cc:538] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:39.702299 21689 tablet_copy_client.cc:323] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:39.705232 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet 8d8e1b8edadd42f08808ffaf11b40c8e from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:39.705871 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet 8d8e1b8edadd42f08808ffaf11b40c8e from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-8d8e1b8edadd42f08808ffaf11b40c8e
I20250116 02:58:39.714465 21691 tablet_bootstrap.cc:492] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:39.720572 20618 tablet_copy_source_session.cc:215] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:39.724465 21689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8e1b8edadd42f08808ffaf11b40c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:39.744316 21691 tablet_bootstrap.cc:492] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:39.745101 21689 tablet_copy_client.cc:806] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:39.745556 21691 tablet_bootstrap.cc:492] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:39.745770 21689 tablet_copy_client.cc:670] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:39.746397 21691 ts_tablet_manager.cc:1397] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.002s
I20250116 02:58:39.749912 21691 raft_consensus.cc:357] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.750993 21691 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:39.751763 21691 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.751581 21689 tablet_copy_client.cc:538] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:39.755131 21691 ts_tablet_manager.cc:1428] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250116 02:58:39.767638 21689 tablet_bootstrap.cc:492] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:39.774279 20766 tablet_copy_service.cc:342] P 17446c8dacf04461acb843547665250f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-99adad9dddaa4121a293aef6fccd1694 received from {username='slave'} at 127.0.0.1:35540
I20250116 02:58:39.775125 20766 tablet_copy_service.cc:434] P 17446c8dacf04461acb843547665250f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-99adad9dddaa4121a293aef6fccd1694 on tablet 99adad9dddaa4121a293aef6fccd1694 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:39.777724 21697 ts_tablet_manager.cc:927] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:39.790464 21697 tablet_copy_client.cc:323] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:39.794143 20692 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 19c9bd4e72374377958093c605e5c6ae from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:39.794787 20692 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 19c9bd4e72374377958093c605e5c6ae from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-19c9bd4e72374377958093c605e5c6ae
I20250116 02:58:39.797883 21689 tablet_bootstrap.cc:492] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:39.799002 21689 tablet_bootstrap.cc:492] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:39.799782 21689 ts_tablet_manager.cc:1397] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.008s
I20250116 02:58:39.802450 21689 raft_consensus.cc:357] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.803483 21689 raft_consensus.cc:738] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:39.803576 20692 tablet_copy_source_session.cc:215] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:39.804262 21689 consensus_queue.cc:260] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.807323 21697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c9bd4e72374377958093c605e5c6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:39.807369 21689 ts_tablet_manager.cc:1428] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20250116 02:58:39.809680 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-8d8e1b8edadd42f08808ffaf11b40c8e received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:39.810165 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-8d8e1b8edadd42f08808ffaf11b40c8e on tablet 8d8e1b8edadd42f08808ffaf11b40c8e with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:39.832227 21697 tablet_copy_client.cc:806] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:39.832814 21697 tablet_copy_client.cc:670] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:39.836921 21697 tablet_copy_client.cc:538] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:39.844868 21697 tablet_bootstrap.cc:492] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:39.870297 21697 tablet_bootstrap.cc:492] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:39.871417 21697 tablet_bootstrap.cc:492] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:39.872218 21697 ts_tablet_manager.cc:1397] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.005s
I20250116 02:58:39.875739 21697 raft_consensus.cc:357] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.876485 21697 raft_consensus.cc:738] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:39.877133 21697 consensus_queue.cc:260] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:39.879940 21697 ts_tablet_manager.cc:1428] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250116 02:58:39.881613 20692 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-19c9bd4e72374377958093c605e5c6ae received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:39.882038 20692 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-19c9bd4e72374377958093c605e5c6ae on tablet 19c9bd4e72374377958093c605e5c6ae with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:40.104051 21334 raft_consensus.cc:1212] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:40.113435 21334 raft_consensus.cc:1212] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:40.259052 21334 raft_consensus.cc:1212] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:40.318217 21334 raft_consensus.cc:1212] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:40.609972 21676 raft_consensus.cc:1059] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:40.613083 21676 consensus_queue.cc:237] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:40.622227 21334 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.623028 20751 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.624265 21651 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:40.626103 21688 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:40.628085 20605 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.629652 21688 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:40.644001 21658 raft_consensus.cc:1059] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:40.647114 21658 consensus_queue.cc:237] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:40.648710 21597 raft_consensus.cc:1059] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:40.651387 21597 consensus_queue.cc:237] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:40.655946 21688 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.686476 20753 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.699304 20607 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.709968 21333 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.724227 21334 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.727717 20503 catalog_manager.cc:5526] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:40.736922 20606 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.739351 21666 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:40.746711 21706 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:40.797225 21333 raft_consensus.cc:1270] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.801469 20753 raft_consensus.cc:1270] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.808748 20677 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.811060 21702 raft_consensus.cc:1059] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:40.814085 21706 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:40.808817 20681 raft_consensus.cc:1270] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.826964 20603 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.826802 21666 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.829218 21658 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:40.831046 21716 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:40.839203 21616 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:40.838860 21702 consensus_queue.cc:237] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:40.898116 20681 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.901933 21713 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.905699 21333 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.907675 20753 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.974862 20607 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.977412 20751 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.979564 21688 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:40.981194 21651 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:40.983645 21336 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:40.974831 21333 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:40.993873 21702 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250116 02:58:40.998373 20681 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.006734 20676 consensus_queue.cc:237] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:41.015183 21676 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.016362 21334 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:41.031919 20502 catalog_manager.cc:5526] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.035063 20753 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:41.036784 21688 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:41.039124 21688 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:41.063505 21334 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.063736 21732 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.073549 20504 catalog_manager.cc:5526] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.089615 20751 raft_consensus.cc:2949] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.099274 20607 consensus_queue.cc:237] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:41.107828 21333 raft_consensus.cc:1270] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:41.110765 21616 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:41.112951 20749 raft_consensus.cc:1270] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:41.125516 21334 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.127493 21716 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:41.129400 20503 catalog_manager.cc:5526] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.121081 20749 consensus_queue.cc:237] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:41.139966 21715 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.156594 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet da8115c09ba24b95a9969edcc6ca4d8c with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:41.160820 20607 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.170974 21333 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:41.172426 20753 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.172331 20503 catalog_manager.cc:5526] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.175872 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet da8115c09ba24b95a9969edcc6ca4d8c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:41.176163 21666 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:41.188017 20605 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:41.191227 21708 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:41.187997 21334 raft_consensus.cc:2949] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.200764 21711 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.208979 20605 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.216317 21734 tablet_replica.cc:331] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:41.217849 21734 raft_consensus.cc:2238] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:41.218873 21734 raft_consensus.cc:2267] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:41.222077 20491 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 99adad9dddaa4121a293aef6fccd1694 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:41.227016 21734 ts_tablet_manager.cc:1905] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:41.229455 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8d8e1b8edadd42f08808ffaf11b40c8e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:41.229780 20505 catalog_manager.cc:5526] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.248546 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8e1b8edadd42f08808ffaf11b40c8e with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:41.256561 21734 ts_tablet_manager.cc:1918] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:41.257298 21734 log.cc:1198] T da8115c09ba24b95a9969edcc6ca4d8c P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/da8115c09ba24b95a9969edcc6ca4d8c
I20250116 02:58:41.259461 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet da8115c09ba24b95a9969edcc6ca4d8c (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:41.263345 20749 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.267107 20502 catalog_manager.cc:5526] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:41.270092 20676 consensus_queue.cc:237] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:41.279803 21735 tablet_replica.cc:331] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:41.284556 21336 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:41.284519 21334 raft_consensus.cc:2949] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.286435 21729 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:41.290970 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet 99adad9dddaa4121a293aef6fccd1694 with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:41.301172 20753 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:41.303830 21676 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.307912 21735 raft_consensus.cc:2238] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:41.309804 21735 raft_consensus.cc:2267] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:41.314203 21735 ts_tablet_manager.cc:1905] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:41.314134 21702 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:41.312568 21336 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.319720 20749 raft_consensus.cc:2949] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:41.326442 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 19c9bd4e72374377958093c605e5c6ae with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:41.335460 21735 ts_tablet_manager.cc:1918] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:41.336205 21735 log.cc:1198] T 8d8e1b8edadd42f08808ffaf11b40c8e P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/8d8e1b8edadd42f08808ffaf11b40c8e
I20250116 02:58:41.342047 21735 tablet_replica.cc:331] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:41.341915 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet 8d8e1b8edadd42f08808ffaf11b40c8e (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:41.339839 20502 catalog_manager.cc:5526] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:41.343298 21735 raft_consensus.cc:2238] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:41.344230 21735 raft_consensus.cc:2267] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:41.350119 21735 ts_tablet_manager.cc:1905] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:41.356009 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c9bd4e72374377958093c605e5c6ae with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:41.357339 21734 tablet_replica.cc:331] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:41.358428 21734 raft_consensus.cc:2238] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:41.359242 21734 raft_consensus.cc:2267] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:41.368352 21734 ts_tablet_manager.cc:1905] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:41.369421 21735 ts_tablet_manager.cc:1918] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:41.381465 21735 log.cc:1198] T 99adad9dddaa4121a293aef6fccd1694 P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/99adad9dddaa4121a293aef6fccd1694
I20250116 02:58:41.383399 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet 99adad9dddaa4121a293aef6fccd1694 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:41.397985 21734 ts_tablet_manager.cc:1918] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:41.398551 21734 log.cc:1198] T 19c9bd4e72374377958093c605e5c6ae P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/19c9bd4e72374377958093c605e5c6ae
I20250116 02:58:41.400782 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet 19c9bd4e72374377958093c605e5c6ae (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:42.406497 20607 consensus_queue.cc:237] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:42.412708 20753 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:42.413127 20676 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:42.413542 20571 consensus_peers.cc:487] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49cb7aba83e046129200b93f32eb3585. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:42.414352 21736 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:42.415819 21715 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:42.424355 21736 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.426676 20753 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.432142 20676 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.439438 20605 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:42.447672 20505 catalog_manager.cc:5526] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:42.461097 20571 consensus_peers.cc:487] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5ac0297b7614c4a8601eded83fff0a2. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:42.462448 20753 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:42.463972 20676 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:42.467420 21715 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:42.469481 21616 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:42.500012 21616 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.502807 20676 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.512898 20753 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.528831 20681 consensus_queue.cc:237] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:42.529026 20503 catalog_manager.cc:5526] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.547618 20752 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:42.548416 20646 consensus_peers.cc:487] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b626139c64c14e309e332bc7605ba6f1. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:42.550125 21702 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:42.552446 20605 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:42.553967 21702 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:42.561221 21729 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.568113 20752 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.570835 20607 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.592259 20505 catalog_manager.cc:5526] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.595497 20681 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:42.627020 20605 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:42.628513 21753 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250116 02:58:42.629768 20646 consensus_peers.cc:487] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0d2558ee3674a5889434a45c66ac3c1. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:42.635360 20753 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:42.638672 21751 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:42.660367 21752 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.662020 20753 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.663905 20605 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:42.683182 20505 catalog_manager.cc:5526] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:42.805892 21757 ts_tablet_manager.cc:927] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:42.808231 21757 tablet_copy_client.cc:323] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:42.810437 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet 49cb7aba83e046129200b93f32eb3585 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:42.811090 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet 49cb7aba83e046129200b93f32eb3585 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-49cb7aba83e046129200b93f32eb3585
I20250116 02:58:42.833992 20618 tablet_copy_source_session.cc:215] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:42.837077 21757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cb7aba83e046129200b93f32eb3585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:42.854660 21757 tablet_copy_client.cc:806] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:42.855330 21757 tablet_copy_client.cc:670] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:42.859107 21757 tablet_copy_client.cc:538] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:42.872660 21757 tablet_bootstrap.cc:492] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:42.897068 21757 tablet_bootstrap.cc:492] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:42.897989 21757 tablet_bootstrap.cc:492] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:42.898710 21757 ts_tablet_manager.cc:1397] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.003s
I20250116 02:58:42.901460 21757 raft_consensus.cc:357] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:42.902693 21757 raft_consensus.cc:738] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:42.903455 21757 consensus_queue.cc:260] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:42.906944 21757 ts_tablet_manager.cc:1428] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250116 02:58:42.909518 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-49cb7aba83e046129200b93f32eb3585 received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:42.910017 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-49cb7aba83e046129200b93f32eb3585 on tablet 49cb7aba83e046129200b93f32eb3585 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:42.969750 21757 ts_tablet_manager.cc:927] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:42.971536 21757 tablet_copy_client.cc:323] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:42.973508 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet b5ac0297b7614c4a8601eded83fff0a2 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:42.974054 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet b5ac0297b7614c4a8601eded83fff0a2 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-b5ac0297b7614c4a8601eded83fff0a2
I20250116 02:58:42.980234 20618 tablet_copy_source_session.cc:215] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:42.983455 21757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ac0297b7614c4a8601eded83fff0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:42.999493 21757 tablet_copy_client.cc:806] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:43.000245 21757 tablet_copy_client.cc:670] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:43.004269 21757 tablet_copy_client.cc:538] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:43.014500 21757 tablet_bootstrap.cc:492] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:43.046409 21757 tablet_bootstrap.cc:492] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:43.047910 21757 tablet_bootstrap.cc:492] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:43.048775 21757 ts_tablet_manager.cc:1397] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.012s
I20250116 02:58:43.054097 21757 raft_consensus.cc:357] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:43.055974 21757 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:43.057310 21757 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:43.064395 21757 ts_tablet_manager.cc:1428] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.015s user 0.010s sys 0.000s
I20250116 02:58:43.066326 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-b5ac0297b7614c4a8601eded83fff0a2 received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:43.066896 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-b5ac0297b7614c4a8601eded83fff0a2 on tablet b5ac0297b7614c4a8601eded83fff0a2 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:43.083578 21757 ts_tablet_manager.cc:927] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:43.102046 21764 ts_tablet_manager.cc:927] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:43.104840 21757 tablet_copy_client.cc:323] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:43.106981 20692 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet b626139c64c14e309e332bc7605ba6f1 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:43.107582 20692 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet b626139c64c14e309e332bc7605ba6f1 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-b626139c64c14e309e332bc7605ba6f1
I20250116 02:58:43.112994 21764 tablet_copy_client.cc:323] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:43.115373 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet b0d2558ee3674a5889434a45c66ac3c1 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:43.116269 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet b0d2558ee3674a5889434a45c66ac3c1 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-b0d2558ee3674a5889434a45c66ac3c1
I20250116 02:58:43.121385 20692 tablet_copy_source_session.cc:215] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:43.125008 21757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b626139c64c14e309e332bc7605ba6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:43.130903 20693 tablet_copy_source_session.cc:215] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:43.134354 21764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d2558ee3674a5889434a45c66ac3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:43.145697 21757 tablet_copy_client.cc:806] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:43.146384 21757 tablet_copy_client.cc:670] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:43.151669 21757 tablet_copy_client.cc:538] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:43.153504 21764 tablet_copy_client.cc:806] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:43.155586 21764 tablet_copy_client.cc:670] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:43.165191 21764 tablet_copy_client.cc:538] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:43.172646 21757 tablet_bootstrap.cc:492] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:43.182535 21764 tablet_bootstrap.cc:492] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:43.216843 21757 tablet_bootstrap.cc:492] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:43.217957 21757 tablet_bootstrap.cc:492] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:43.218986 21757 ts_tablet_manager.cc:1397] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.047s user 0.025s sys 0.020s
I20250116 02:58:43.222537 21757 raft_consensus.cc:357] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:43.223659 21757 raft_consensus.cc:738] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:43.224306 21757 consensus_queue.cc:260] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:43.230767 21757 ts_tablet_manager.cc:1428] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250116 02:58:43.232896 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-b626139c64c14e309e332bc7605ba6f1 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:43.233409 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-b626139c64c14e309e332bc7605ba6f1 on tablet b626139c64c14e309e332bc7605ba6f1 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:43.254876 21764 tablet_bootstrap.cc:492] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:43.256422 21764 tablet_bootstrap.cc:492] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:43.257122 21764 ts_tablet_manager.cc:1397] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.075s user 0.028s sys 0.000s
I20250116 02:58:43.260038 21764 raft_consensus.cc:357] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:43.260989 21764 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:43.261664 21764 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:43.264791 21764 ts_tablet_manager.cc:1428] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250116 02:58:43.268589 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-b0d2558ee3674a5889434a45c66ac3c1 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:43.269178 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-b0d2558ee3674a5889434a45c66ac3c1 on tablet b0d2558ee3674a5889434a45c66ac3c1 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:43.311306 21334 raft_consensus.cc:1212] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:43.455199 21334 raft_consensus.cc:1212] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:43.498890 21334 raft_consensus.cc:1212] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:43.624820 21334 raft_consensus.cc:1212] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:43.899812 21770 raft_consensus.cc:1059] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:43.905736 21770 consensus_queue.cc:237] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:43.909585 21751 raft_consensus.cc:1059] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:43.913319 21751 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:43.929814 20752 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:43.935948 21336 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:43.936866 21752 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:43.937933 20605 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:43.939298 21774 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:43.940204 21752 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:43.951468 20753 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:43.952797 20681 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:43.953970 21334 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:43.955564 21769 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:43.957041 21770 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:43.961001 21715 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:43.980863 21753 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:43.982538 21333 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.002353 20502 catalog_manager.cc:5526] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.101606 21332 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.117301 20681 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 02:58:44.133927 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 67714.8us
I20250116 02:58:44.093149 21715 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.146062 21751 raft_consensus.cc:1059] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:44.148967 21751 consensus_queue.cc:237] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:58:44.154487 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 72839us
I20250116 02:58:44.156869 20753 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.203836 21770 raft_consensus.cc:1059] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:44.210680 20752 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.226246 21332 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:44.228036 21751 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:44.227236 21770 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:44.252198 20607 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.254367 20681 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:44.255959 20752 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:44.257454 21769 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:44.259296 21791 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:44.260597 21333 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:44.262265 21791 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250116 02:58:44.258527 20505 catalog_manager.cc:5526] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:44.267516 20605 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:44.290194 20753 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:44.301105 21751 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:44.308853 21792 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:44.323904 21791 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.327456 20681 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.372386 20749 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.389117 20606 consensus_queue.cc:237] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:44.402169 21794 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.403610 20749 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.400028 20505 catalog_manager.cc:5526] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.416812 20605 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.427460 21332 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.429814 21333 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.447804 20503 catalog_manager.cc:5526] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.457931 20681 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:44.462316 21332 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:44.468145 21782 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:44.472630 21782 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:44.473451 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:44.483458 20606 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:44.489614 21715 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.493022 21332 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:44.494812 21769 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:44.494536 21336 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.506451 20677 consensus_queue.cc:237] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:44.506942 20676 raft_consensus.cc:2949] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.512389 20749 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:44.512907 21769 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.512948 20681 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:44.517021 21333 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.523810 21791 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:44.524850 20607 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:44.526477 21792 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:44.529000 21332 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:44.533278 20749 raft_consensus.cc:2949] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.534767 21753 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:44.542496 21751 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.546990 21335 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.550179 21334 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:44.553143 21753 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:44.574445 21786 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.580157 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 49cb7aba83e046129200b93f32eb3585 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:44.589229 21334 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.591886 20502 catalog_manager.cc:5526] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:44.608804 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b626139c64c14e309e332bc7605ba6f1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:44.611166 20749 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:44.615006 21783 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:44.620965 20605 raft_consensus.cc:2949] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.624207 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cb7aba83e046129200b93f32eb3585 with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:44.637071 20752 raft_consensus.cc:2949] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 02:58:44.674223 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 61669.2us
I20250116 02:58:44.687734 20502 catalog_manager.cc:5526] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:44.554795 20504 catalog_manager.cc:5526] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.723381 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet b626139c64c14e309e332bc7605ba6f1 with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
W20250116 02:58:44.744204 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 130504us
I20250116 02:58:44.709672 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b0d2558ee3674a5889434a45c66ac3c1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:44.706427 21807 tablet_replica.cc:331] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:44.752803 21807 raft_consensus.cc:2238] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:44.754993 21807 raft_consensus.cc:2267] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:44.753600 20605 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:44.757652 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ac0297b7614c4a8601eded83fff0a2 with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:44.758270 20605 raft_consensus.cc:604] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:44.760916 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b5ac0297b7614c4a8601eded83fff0a2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:44.768798 20505 catalog_manager.cc:5526] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:44.770238 21807 ts_tablet_manager.cc:1905] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:44.771430 21812 tablet_replica.cc:331] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:44.786480 21812 raft_consensus.cc:2238] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:44.790679 21812 raft_consensus.cc:2267] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:44.796377 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d2558ee3674a5889434a45c66ac3c1 with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
W20250116 02:58:44.799320 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 54164.8us
I20250116 02:58:44.819820 21812 ts_tablet_manager.cc:1905] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:44.828033 21814 tablet_replica.cc:331] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:44.829339 21814 raft_consensus.cc:2238] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:44.830211 21807 ts_tablet_manager.cc:1918] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:44.830348 21814 raft_consensus.cc:2267] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:44.830983 21807 log.cc:1198] T 49cb7aba83e046129200b93f32eb3585 P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/49cb7aba83e046129200b93f32eb3585
I20250116 02:58:44.836920 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet 49cb7aba83e046129200b93f32eb3585 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:44.841881 21814 ts_tablet_manager.cc:1905] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:44.836954 21807 tablet_replica.cc:331] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:44.846354 21807 raft_consensus.cc:2238] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:44.855118 21807 raft_consensus.cc:2267] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:44.858842 21807 ts_tablet_manager.cc:1905] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:44.863310 21812 ts_tablet_manager.cc:1918] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:44.863930 21812 log.cc:1198] T b5ac0297b7614c4a8601eded83fff0a2 P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/b5ac0297b7614c4a8601eded83fff0a2
I20250116 02:58:44.865840 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet b5ac0297b7614c4a8601eded83fff0a2 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:44.867751 20749 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:44.868435 20749 raft_consensus.cc:604] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:44.877343 21807 ts_tablet_manager.cc:1918] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:44.877912 21807 log.cc:1198] T b626139c64c14e309e332bc7605ba6f1 P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/b626139c64c14e309e332bc7605ba6f1
I20250116 02:58:44.879635 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet b626139c64c14e309e332bc7605ba6f1 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:44.884315 21811 raft_consensus.cc:988] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:44.884909 21811 raft_consensus.cc:1076] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:44.889649 21336 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56674
I20250116 02:58:44.890184 21336 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:44.890566 21336 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:44.892402 21814 ts_tablet_manager.cc:1918] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:44.892853 21814 log.cc:1198] T b0d2558ee3674a5889434a45c66ac3c1 P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/b0d2558ee3674a5889434a45c66ac3c1
I20250116 02:58:44.894354 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet b0d2558ee3674a5889434a45c66ac3c1 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:44.895766 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d3fee7b273345558c661eda0710029c"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:44.896399 20681 raft_consensus.cc:604] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:44.897569 21336 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:44.899089 20749 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:44.899852 20749 raft_consensus.cc:604] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:44.900820 21336 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:44.901820 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:44.902186 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:44.902393 20677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a569f56f9d2143869b29c78321047d00"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:44.902555 20681 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:44.903126 20749 raft_consensus.cc:3049] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:44.903211 20677 raft_consensus.cc:604] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:44.903461 20749 raft_consensus.cc:738] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:44.904158 20749 consensus_queue.cc:260] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:44.905974 20749 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:44.906109 20752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:44.906769 20752 raft_consensus.cc:604] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:44.908205 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250116 02:58:44.909816 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:44.910513 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:44.912092 20749 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:44.913426 20681 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:44.913602 21300 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:44.914791 21761 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:44.915936 21761 raft_consensus.cc:695] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:44.917033 21761 consensus_queue.cc:237] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:44.946142 20504 catalog_manager.cc:5526] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:44.984959 21769 raft_consensus.cc:988] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:44.985471 21782 raft_consensus.cc:1076] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:44.987223 21336 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56662
I20250116 02:58:44.987928 21336 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:44.988296 21336 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:44.995769 21336 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:44.997967 21336 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:44.998701 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:44.999469 20681 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:44.998658 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:45.000416 20607 raft_consensus.cc:3049] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:45.000703 20607 raft_consensus.cc:738] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:45.001495 20607 consensus_queue.cc:260] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:45.002915 20607 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.006906 20681 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:45.008301 20607 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:45.009356 21299 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:45.011623 21761 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:45.012144 21761 raft_consensus.cc:695] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:45.013036 21761 consensus_queue.cc:237] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:45.021688 20504 catalog_manager.cc:5526] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:45.057549 21784 raft_consensus.cc:988] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:45.058107 21811 raft_consensus.cc:1076] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:45.059705 21335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56674
I20250116 02:58:45.060246 21335 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:45.060564 21335 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.066767 21335 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:45.070629 21335 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:45.070823 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:45.073657 20752 raft_consensus.cc:3049] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:45.074051 20752 raft_consensus.cc:738] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:45.074537 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:45.074926 20752 consensus_queue.cc:260] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:45.076733 20752 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.078832 20607 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.084187 20752 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:45.085443 21300 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:45.085740 20607 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:45.086391 21761 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:45.086931 21761 raft_consensus.cc:695] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:45.087929 21761 consensus_queue.cc:237] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:45.096544 20504 catalog_manager.cc:5526] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:45.252228 21810 raft_consensus.cc:988] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:45.252827 21811 raft_consensus.cc:1076] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:45.255045 21335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56674
I20250116 02:58:45.258594 21335 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:45.259064 21335 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.267333 21335 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:45.269397 21335 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:45.270509 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:45.271356 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:45.272363 20607 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.271268 20752 raft_consensus.cc:3049] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:45.272866 20752 raft_consensus.cc:738] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:45.273607 20752 consensus_queue.cc:260] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:45.274917 20752 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.283046 20607 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:45.282711 20752 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:45.284511 21300 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:45.285809 21761 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:45.286406 21761 raft_consensus.cc:695] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:45.288079 21761 consensus_queue.cc:237] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:45.300938 20504 catalog_manager.cc:5526] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:45.316349 21752 raft_consensus.cc:988] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:45.316900 21786 raft_consensus.cc:1076] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:45.318261 20752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d3fee7b273345558c661eda0710029c"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:45.318843 20752 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:45.319118 20752 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.326452 20752 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:45.328724 20752 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:45.329747 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:45.329596 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:45.330322 20681 raft_consensus.cc:3049] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:45.330488 20605 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.330600 20681 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:45.331521 20681 consensus_queue.cc:260] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:45.332690 20681 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.337786 20681 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:45.338061 20605 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:45.338732 20720 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:45.339565 21810 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:45.340054 21810 raft_consensus.cc:695] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:45.340876 21810 consensus_queue.cc:237] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:45.351116 20504 catalog_manager.cc:5526] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:45.374922 21786 raft_consensus.cc:988] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:45.375491 21752 raft_consensus.cc:1076] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:45.376942 20752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a569f56f9d2143869b29c78321047d00"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:45.377504 20752 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:45.377856 20752 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.386495 20752 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:45.388315 20752 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:45.389562 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:45.390223 20605 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.390342 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:45.391260 20681 raft_consensus.cc:3049] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:45.391636 20681 raft_consensus.cc:738] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:45.392405 20681 consensus_queue.cc:260] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:45.393687 20681 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:45.398547 20605 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:45.400846 20723 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:45.401715 20681 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:45.401849 21810 raft_consensus.cc:2798] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:45.402969 21810 raft_consensus.cc:695] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:45.403860 21810 consensus_queue.cc:237] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:45.416496 20504 catalog_manager.cc:5526] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:45.446038 20751 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:45.447435 21761 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:45.460975 20681 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:45.465356 21825 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:45.515436 20681 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:45.517845 21825 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:45.530704 20605 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:45.532083 21816 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:45.604789 20607 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:45.606223 21816 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:45.615697 20751 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:45.617928 21825 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:45.759169 20605 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:45.760444 21825 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:45.769750 20751 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:45.771663 21816 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:45.802896 20676 raft_consensus.cc:1270] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:45.804414 21810 consensus_queue.cc:1035] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:45.846761 20605 raft_consensus.cc:1270] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:45.849033 21784 consensus_queue.cc:1035] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:45.872339 20676 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:45.874048 21784 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250116 02:58:45.888034 20605 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:45.893411 21810 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:45.899622 20751 consensus_queue.cc:237] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:45.917482 20607 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:45.919009 20676 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:45.920724 21810 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250116 02:58:45.922323 20720 consensus_peers.cc:487] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb87db38c5904a70bb4b4b6d11afa1ca. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:45.926996 21810 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:45.932727 21784 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:45.934568 20605 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:45.961982 20605 consensus_queue.cc:237] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:45.964555 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:45.963999 20504 catalog_manager.cc:5526] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:45.987780 20571 consensus_peers.cc:487] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d92adda8583d41508be4e5a8002cac4d. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:45.991747 20677 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:45.993717 21830 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:46.007851 20751 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:46.011111 20676 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:46.012898 21782 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:46.022087 21830 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:46.036689 20676 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:46.074550 20676 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:46.080062 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:46.080869 20681 raft_consensus.cc:604] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:46.085706 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:46.086472 20681 raft_consensus.cc:604] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:58:46.087317 20646 consensus_peers.cc:487] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ed2f7240f574d40b93ad33069e1a632. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:46.089171 21792 raft_consensus.cc:988] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:46.089725 21792 raft_consensus.cc:1076] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:46.091970 21335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56664
I20250116 02:58:46.092772 21335 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:46.093276 21335 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.099138 20605 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:46.100808 21752 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:46.099179 20607 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:46.102983 20607 raft_consensus.cc:604] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:46.097309 20505 catalog_manager.cc:5526] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:46.110968 20751 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:46.111845 20751 raft_consensus.cc:604] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:46.117585 20751 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a569f56f9d2143869b29c78321047d00"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:46.116401 21335 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:46.118175 20751 raft_consensus.cc:604] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:46.121217 20677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:46.121544 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:46.122043 20677 raft_consensus.cc:604] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:46.122310 20676 raft_consensus.cc:3049] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:46.122643 20676 raft_consensus.cc:738] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:46.124231 20752 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:46.119768 21335 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:46.128463 21752 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:46.137851 20757 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:46.141038 20676 consensus_queue.cc:260] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:46.139230 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:46.142642 20676 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.153748 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250116 02:58:46.148690 20751 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.157855 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:46.157034 21752 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:46.158900 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250116 02:58:46.197337 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 68473.4us
I20250116 02:58:46.199878 20605 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:46.206897 20676 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:46.210233 20757 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:46.216815 20554 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20250116 02:58:46.218690 20504 catalog_manager.cc:5526] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:46.228295 20751 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:46.230485 21300 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:46.234663 21816 raft_consensus.cc:2798] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:46.235219 21816 raft_consensus.cc:695] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:46.236121 21816 consensus_queue.cc:237] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:46.252862 21752 raft_consensus.cc:988] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
W20250116 02:58:46.253058 20554 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 31db17a68ff141f3bd799ccc59c81fcc, TS f65e8d7f76144fab9725ca32a6328c3f -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:46.253511 20554 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 31db17a68ff141f3bd799ccc59c81fcc, TS f65e8d7f76144fab9725ca32a6328c3f -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.255218 21752 raft_consensus.cc:1076] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:46.256778 21336 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56664
I20250116 02:58:46.257393 21336 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:46.257767 21336 raft_consensus.cc:3054] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.272185 21336 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:46.277392 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:46.277244 21336 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:46.278091 20605 raft_consensus.cc:3054] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.278276 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:46.278973 20677 raft_consensus.cc:3049] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:46.279271 20677 raft_consensus.cc:738] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:46.280094 20677 consensus_queue.cc:260] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:46.281322 20677 raft_consensus.cc:3054] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.279269 20504 catalog_manager.cc:5526] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:46.288873 20677 raft_consensus.cc:2463] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:46.294031 21299 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:46.295210 21816 raft_consensus.cc:2798] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:46.296090 21816 raft_consensus.cc:695] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:46.297209 21810 raft_consensus.cc:988] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:46.297082 21816 consensus_queue.cc:237] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:46.297755 21784 raft_consensus.cc:1076] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:46.299490 20605 raft_consensus.cc:2463] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:46.301177 21336 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56674
I20250116 02:58:46.301828 21336 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:46.302246 21336 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.310534 20504 catalog_manager.cc:5526] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:46.322801 21336 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:46.340313 21336 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:46.341310 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:46.341924 20751 raft_consensus.cc:3049] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:46.342198 20751 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:46.342901 20751 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:46.343945 20751 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.344669 21811 raft_consensus.cc:988] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:46.345129 21811 raft_consensus.cc:1076] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:46.347064 20607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a569f56f9d2143869b29c78321047d00"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43850
I20250116 02:58:46.347642 20607 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:46.347999 20607 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:46.347123 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:46.349031 20605 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.360841 20605 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:46.362309 21859 ts_tablet_manager.cc:927] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:46.362195 21302 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:46.363384 21816 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:46.363915 21816 raft_consensus.cc:695] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:46.364738 21816 consensus_queue.cc:237] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:46.365862 20751 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:46.363379 20607 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.379496 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:46.380142 20681 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:46.380311 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:46.381042 20751 raft_consensus.cc:3049] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:58:46.381346 20751 raft_consensus.cc:738] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:46.382133 20751 consensus_queue.cc:260] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.383390 20751 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:46.383790 20607 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:46.386722 21859 tablet_copy_client.cc:323] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.131:43647
I20250116 02:58:46.391211 20766 tablet_copy_service.cc:140] P 17446c8dacf04461acb843547665250f: Received BeginTabletCopySession request for tablet eb87db38c5904a70bb4b4b6d11afa1ca from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:35540)
I20250116 02:58:46.392071 20766 tablet_copy_service.cc:161] P 17446c8dacf04461acb843547665250f: Beginning new tablet copy session on tablet eb87db38c5904a70bb4b4b6d11afa1ca from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:35540: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-eb87db38c5904a70bb4b4b6d11afa1ca
I20250116 02:58:46.393501 20751 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:58:46.395013 20572 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:46.398125 21851 raft_consensus.cc:2798] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:58:46.398813 21851 raft_consensus.cc:695] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:46.399266 20681 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:58:46.399819 21851 consensus_queue.cc:237] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.410650 20766 tablet_copy_source_session.cc:215] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:46.413246 20504 catalog_manager.cc:5526] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:46.414690 21859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb87db38c5904a70bb4b4b6d11afa1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:46.425019 20503 catalog_manager.cc:5526] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 2 to 3, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 3 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:46.439577 21859 tablet_copy_client.cc:806] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:46.440215 21859 tablet_copy_client.cc:670] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:46.446111 21862 ts_tablet_manager.cc:927] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:46.446274 21859 tablet_copy_client.cc:538] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:46.448531 21862 tablet_copy_client.cc:323] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:46.450652 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet d92adda8583d41508be4e5a8002cac4d from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:46.451315 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet d92adda8583d41508be4e5a8002cac4d from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-d92adda8583d41508be4e5a8002cac4d
I20250116 02:58:46.459044 21859 tablet_bootstrap.cc:492] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:46.462208 20618 tablet_copy_source_session.cc:215] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:46.466931 21862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92adda8583d41508be4e5a8002cac4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:46.481690 21859 tablet_bootstrap.cc:492] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:46.483160 21859 tablet_bootstrap.cc:492] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:46.485446 21859 ts_tablet_manager.cc:1397] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.006s
I20250116 02:58:46.490236 21859 raft_consensus.cc:357] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:46.492908 21859 raft_consensus.cc:738] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:46.493829 21859 consensus_queue.cc:260] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:46.496696 21859 ts_tablet_manager.cc:1428] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20250116 02:58:46.497610 21862 tablet_copy_client.cc:806] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:46.498092 21862 tablet_copy_client.cc:670] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:46.499505 20766 tablet_copy_service.cc:342] P 17446c8dacf04461acb843547665250f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-eb87db38c5904a70bb4b4b6d11afa1ca received from {username='slave'} at 127.0.0.1:35540
I20250116 02:58:46.500056 20766 tablet_copy_service.cc:434] P 17446c8dacf04461acb843547665250f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-eb87db38c5904a70bb4b4b6d11afa1ca on tablet eb87db38c5904a70bb4b4b6d11afa1ca with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:46.507563 21862 tablet_copy_client.cc:538] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:46.517428 21862 tablet_bootstrap.cc:492] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:46.521718 21769 raft_consensus.cc:988] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:46.522392 21769 raft_consensus.cc:1076] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:46.526882 20751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50646
I20250116 02:58:46.527566 20751 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:46.528016 20751 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.540186 20751 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.542254 20751 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:46.544376 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:46.545444 20607 raft_consensus.cc:3049] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:46.545878 20607 raft_consensus.cc:738] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:46.546864 20607 consensus_queue.cc:260] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.547276 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:46.548087 20676 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.554987 20607 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.565419 20676 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:46.570226 21854 raft_consensus.cc:988] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:46.575256 21752 raft_consensus.cc:1076] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:46.575965 20720 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:46.576982 21838 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:46.577492 21838 raft_consensus.cc:695] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:46.578426 21838 consensus_queue.cc:237] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.574095 20607 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:46.602480 20757 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:46.603235 20757 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:46.603650 20757 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.610033 20503 catalog_manager.cc:5526] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:46.616627 20757 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.631569 20757 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:46.634137 20749 raft_consensus.cc:1270] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:46.635988 21816 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:46.637591 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:46.639283 20676 raft_consensus.cc:3049] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:46.639739 20676 raft_consensus.cc:738] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:46.640739 20676 consensus_queue.cc:260] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.643986 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:46.644793 20607 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.647253 20676 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:46.663106 21859 ts_tablet_manager.cc:927] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:46.668138 21862 tablet_bootstrap.cc:492] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:46.668448 21859 tablet_copy_client.cc:323] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:46.669261 21862 tablet_bootstrap.cc:492] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:46.670053 21862 ts_tablet_manager.cc:1397] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.153s user 0.070s sys 0.065s
I20250116 02:58:46.672473 20676 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:46.675873 21862 raft_consensus.cc:357] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:46.676894 21862 raft_consensus.cc:738] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:46.677549 21862 consensus_queue.cc:260] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:46.679103 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 0ed2f7240f574d40b93ad33069e1a632 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:46.679673 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 0ed2f7240f574d40b93ad33069e1a632 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-0ed2f7240f574d40b93ad33069e1a632
I20250116 02:58:46.680267 21862 ts_tablet_manager.cc:1428] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20250116 02:58:46.675916 20607 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:46.682906 20723 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:46.684594 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-d92adda8583d41508be4e5a8002cac4d received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:46.685235 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-d92adda8583d41508be4e5a8002cac4d on tablet d92adda8583d41508be4e5a8002cac4d with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:46.691386 21838 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:46.692054 21838 raft_consensus.cc:695] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:46.692992 21838 consensus_queue.cc:237] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:46.711438 20676 raft_consensus.cc:1270] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:46.711542 20693 tablet_copy_source_session.cc:215] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:46.715373 21872 consensus_queue.cc:1035] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:46.716377 20605 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:46.716800 21859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ed2f7240f574d40b93ad33069e1a632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:46.720230 21872 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:46.726980 21336 raft_consensus.cc:1212] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:46.733557 20504 catalog_manager.cc:5526] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:46.818578 21859 tablet_copy_client.cc:806] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:46.821138 21859 tablet_copy_client.cc:670] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:46.829157 20681 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:46.840030 21872 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:46.854434 21859 tablet_copy_client.cc:538] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:46.866729 21859 tablet_bootstrap.cc:492] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:46.896555 20681 raft_consensus.cc:1270] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:46.898188 21830 consensus_queue.cc:1035] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:46.902452 21859 tablet_bootstrap.cc:492] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:46.903708 21859 tablet_bootstrap.cc:492] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:46.906113 21859 ts_tablet_manager.cc:1397] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.014s
I20250116 02:58:46.910290 21859 raft_consensus.cc:357] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:46.911311 21859 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:46.912012 21859 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:46.914979 21859 ts_tablet_manager.cc:1428] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250116 02:58:46.916734 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-0ed2f7240f574d40b93ad33069e1a632 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:46.917188 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-0ed2f7240f574d40b93ad33069e1a632 on tablet 0ed2f7240f574d40b93ad33069e1a632 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:46.925974 21336 raft_consensus.cc:1212] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:46.930367 20751 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:46.931885 21825 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:46.965935 20749 raft_consensus.cc:1270] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:46.981794 20605 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:46.984645 21825 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:46.982156 21851 consensus_queue.cc:1035] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250116 02:58:47.019157 20606 raft_consensus.cc:1270] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:47.021551 21811 consensus_queue.cc:1035] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:47.068059 20677 raft_consensus.cc:1270] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:47.076812 21811 consensus_queue.cc:1035] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:47.174136 21784 raft_consensus.cc:1059] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:47.174599 20676 raft_consensus.cc:1270] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:47.176569 21810 consensus_queue.cc:1035] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:47.176808 21784 consensus_queue.cc:237] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.184063 20605 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:47.185279 21335 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:47.185680 21784 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:47.188589 20681 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:47.196645 21784 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:47.197840 21810 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:47.212690 20606 raft_consensus.cc:1270] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:47.215946 21784 consensus_queue.cc:1035] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:47.242182 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:47.284747 21810 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.290398 21336 raft_consensus.cc:1212] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:47.291312 21334 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.289580 20677 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.311889 20605 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.333318 20503 catalog_manager.cc:5526] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.353219 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:47.353942 20681 raft_consensus.cc:604] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:47.358242 20605 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:47.358897 20605 raft_consensus.cc:604] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:47.361654 20751 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:47.362357 20751 raft_consensus.cc:604] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:47.368067 20751 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:47.368774 20751 raft_consensus.cc:604] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:47.375392 20677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:47.376072 20677 raft_consensus.cc:604] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:47.384735 20751 consensus_queue.cc:237] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.390692 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250116 02:58:47.392644 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:47.393412 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:47.394572 21334 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:47.396378 21838 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:47.401639 20681 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:47.403743 21838 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:47.405421 21811 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.410389 21335 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.422880 20491 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet eb87db38c5904a70bb4b4b6d11afa1ca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:47.428380 21851 raft_consensus.cc:1059] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:47.432303 21851 consensus_queue.cc:237] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.436502 20677 raft_consensus.cc:2949] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.437422 20505 catalog_manager.cc:5526] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.456940 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet eb87db38c5904a70bb4b4b6d11afa1ca with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:47.461369 21898 tablet_replica.cc:331] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:47.462527 21898 raft_consensus.cc:2238] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:47.474478 21898 raft_consensus.cc:2267] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:47.480074 21335 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:47.480427 20757 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:47.482674 20681 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:47.482939 21898 ts_tablet_manager.cc:1905] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:47.483803 21769 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:47.484910 21885 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:47.486059 21851 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:47.518289 21898 ts_tablet_manager.cc:1918] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:47.519019 21898 log.cc:1198] T eb87db38c5904a70bb4b4b6d11afa1ca P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/eb87db38c5904a70bb4b4b6d11afa1ca
I20250116 02:58:47.518118 21851 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.521019 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet eb87db38c5904a70bb4b4b6d11afa1ca (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:47.522449 21335 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.523918 20681 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.547814 20504 catalog_manager.cc:5526] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.563642 21786 raft_consensus.cc:988] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:47.564139 21752 raft_consensus.cc:1076] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:47.571377 20749 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.571405 20757 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:47.575668 20757 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:47.576021 20757 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.585079 20757 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:47.588114 20757 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:47.590850 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:47.591539 20676 raft_consensus.cc:3049] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:47.591852 20676 raft_consensus.cc:738] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:47.592564 20676 consensus_queue.cc:260] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:47.593945 20676 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.598101 21786 raft_consensus.cc:988] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:47.598755 21786 raft_consensus.cc:1076] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:47.601166 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:47.601383 20605 consensus_queue.cc:237] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.602005 20606 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.611713 20751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:47.612453 20751 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:47.612813 20751 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.618927 20757 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:47.621552 20606 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:47.622176 21334 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:47.623569 20723 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:47.624244 21886 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:47.625713 21784 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:47.626356 21784 raft_consensus.cc:695] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:47.627290 21784 consensus_queue.cc:237] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:47.645650 21851 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:47.647488 20676 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:47.664337 21886 raft_consensus.cc:988] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:47.663435 21830 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.664932 21886 raft_consensus.cc:1076] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:47.671028 20751 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.671495 20757 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.673482 20751 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:47.674290 20749 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50646
I20250116 02:58:47.674913 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:47.675550 21334 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.674942 20749 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:47.677786 20749 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.678025 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:47.678704 20676 raft_consensus.cc:3049] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:47.679020 20676 raft_consensus.cc:738] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:47.679798 20676 consensus_queue.cc:260] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.680979 20676 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.688351 21335 raft_consensus.cc:2949] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.693783 20749 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.695322 20676 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:47.697633 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d92adda8583d41508be4e5a8002cac4d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:47.698158 20720 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:47.699210 21811 raft_consensus.cc:2798] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:47.699548 21334 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:47.699805 21811 raft_consensus.cc:695] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:47.701081 21811 consensus_queue.cc:237] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.702844 20749 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:47.705375 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:47.706117 20603 raft_consensus.cc:3049] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:47.706410 20603 raft_consensus.cc:738] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:47.706457 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:47.707311 21335 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.707484 20504 catalog_manager.cc:5526] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:47.709133 20505 catalog_manager.cc:5526] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.708693 20603 consensus_queue.cc:260] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.714059 20603 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.717227 21335 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:47.718376 20720 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:47.723186 20603 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:47.724110 21784 raft_consensus.cc:2798] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:47.724665 21784 raft_consensus.cc:695] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:47.725677 21784 consensus_queue.cc:237] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.737221 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet d92adda8583d41508be4e5a8002cac4d with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:47.739842 21909 tablet_replica.cc:331] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:47.752422 21909 raft_consensus.cc:2238] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:47.753427 21909 raft_consensus.cc:2267] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:47.757855 21909 ts_tablet_manager.cc:1905] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:47.752679 20502 catalog_manager.cc:5526] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:47.760272 21838 raft_consensus.cc:988] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:47.760821 21810 raft_consensus.cc:1076] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:47.760200 20502 catalog_manager.cc:5526] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:47.763569 21335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56674
I20250116 02:58:47.764271 21335 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:47.764693 21335 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.777208 21909 ts_tablet_manager.cc:1918] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:47.777904 21909 log.cc:1198] T d92adda8583d41508be4e5a8002cac4d P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/d92adda8583d41508be4e5a8002cac4d
I20250116 02:58:47.780377 21335 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.781376 21810 raft_consensus.cc:988] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:47.781921 21810 raft_consensus.cc:1076] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:47.783486 21335 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:47.784597 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:47.785152 20749 raft_consensus.cc:3049] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:47.785382 20749 raft_consensus.cc:738] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:47.786015 20749 consensus_queue.cc:260] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.788447 20603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43850
I20250116 02:58:47.789117 20603 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:47.789458 20603 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.790930 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:47.791574 20605 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.787067 20749 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.782874 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet d92adda8583d41508be4e5a8002cac4d (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:47.803103 20605 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:47.799381 20603 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:47.804340 21302 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:47.805261 21816 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:47.805745 21816 raft_consensus.cc:695] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:47.806361 20749 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:47.806684 21816 consensus_queue.cc:237] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.808343 20603 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:47.809100 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:47.809791 20681 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.812615 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:47.813325 20749 raft_consensus.cc:3049] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:47.813617 20749 raft_consensus.cc:738] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:47.814407 20749 consensus_queue.cc:260] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:47.818205 20749 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:47.818840 20681 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:47.820521 20571 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:47.821604 21830 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:47.822091 21830 raft_consensus.cc:695] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:47.823611 21830 consensus_queue.cc:237] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:47.850360 20505 catalog_manager.cc:5526] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:47.866918 20502 catalog_manager.cc:5526] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:47.882483 21903 raft_consensus.cc:1059] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:47.885450 20749 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:47.885432 21903 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.892922 20749 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:47.892925 21335 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:47.895282 20603 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:47.895113 21904 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:47.901022 21903 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:47.903666 21752 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:47.914872 21904 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.917289 20603 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.918812 20749 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.938035 21335 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.938220 20502 catalog_manager.cc:5526] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.956851 20681 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:47.969174 21334 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:47.971908 21855 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:47.975278 20603 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:47.977110 21904 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:47.981351 21903 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.985280 21335 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.990451 20605 raft_consensus.cc:2949] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:47.996712 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 0ed2f7240f574d40b93ad33069e1a632 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:48.002445 20504 catalog_manager.cc:5526] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:48.031773 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed2f7240f574d40b93ad33069e1a632 with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:48.044569 21916 tablet_replica.cc:331] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:48.045367 21916 raft_consensus.cc:2238] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:48.045948 21916 raft_consensus.cc:2267] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:48.049407 21916 ts_tablet_manager.cc:1905] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:48.068508 21916 ts_tablet_manager.cc:1918] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:48.068884 21916 log.cc:1198] T 0ed2f7240f574d40b93ad33069e1a632 P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/0ed2f7240f574d40b93ad33069e1a632
I20250116 02:58:48.070405 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet 0ed2f7240f574d40b93ad33069e1a632 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:48.093303 20677 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:48.094763 21784 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:48.106185 20603 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:48.107800 21784 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:48.127193 20681 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:48.128805 21784 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20250116 02:58:48.179935 21335 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:48.181713 21784 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:48.223405 21335 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:48.225230 21811 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:48.235625 20603 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:48.236984 21784 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:48.264007 20681 raft_consensus.cc:1270] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:48.265511 21851 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:48.275748 20749 raft_consensus.cc:1270] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:48.279220 21830 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:48.353883 20749 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:48.355186 21816 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:48.373160 20605 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:48.375347 21932 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:48.469614 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:48.509954 20752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:48.510516 20752 raft_consensus.cc:604] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:48.513181 20752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:48.513808 20752 raft_consensus.cc:604] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS 231d91433a404468a44ea0a107345aeb
I20250116 02:58:48.516969 20603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38583cae882144f48cec12c72ba697cf"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:48.517637 21810 raft_consensus.cc:988] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f: : Instructing follower 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:48.517695 20603 raft_consensus.cc:604] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 231d91433a404468a44ea0a107345aeb
I20250116 02:58:48.518319 21810 raft_consensus.cc:1076] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:48.520738 20681 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:34870
I20250116 02:58:48.521412 20681 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:48.521799 20681 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.523314 20752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:48.523995 20752 raft_consensus.cc:604] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:48.526937 20677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:48.527558 20677 raft_consensus.cc:604] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:48.528728 20681 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:48.529367 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250116 02:58:48.530395 20681 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:48.531270 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:48.531914 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:48.532049 20603 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.532580 20752 raft_consensus.cc:3049] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:48.532918 20752 raft_consensus.cc:738] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:48.533504 20752 consensus_queue.cc:260] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:48.532104 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:48.535281 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:48.534510 20752 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.539660 20603 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:48.540824 20752 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:48.541152 20649 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:48.542309 21904 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:48.542739 21904 raft_consensus.cc:695] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:48.543715 21904 consensus_queue.cc:237] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:48.569597 20504 catalog_manager.cc:5526] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:48.638912 21810 raft_consensus.cc:988] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:48.639504 21810 raft_consensus.cc:1076] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:48.641053 20603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43850
I20250116 02:58:48.641613 20603 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:48.641876 20603 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.646728 20603 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:48.648547 20603 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:48.649698 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:48.650468 20677 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.651144 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:48.651793 20752 raft_consensus.cc:3049] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:48.652096 20752 raft_consensus.cc:738] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:48.652922 20752 consensus_queue.cc:260] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:48.654353 20752 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.658749 20677 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:48.660295 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:48.661121 21886 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:48.661533 21886 raft_consensus.cc:695] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:48.662495 21886 consensus_queue.cc:237] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:48.663008 20752 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:48.677438 20505 catalog_manager.cc:5526] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:48.741146 21811 raft_consensus.cc:988] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:48.741619 21810 raft_consensus.cc:1076] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:48.743180 21335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56674
I20250116 02:58:48.743845 21335 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:48.744199 21335 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.750829 21335 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:48.752703 21335 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:48.753724 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:48.753790 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:48.754469 20677 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.754580 20749 raft_consensus.cc:3049] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:48.755050 20749 raft_consensus.cc:738] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:48.755822 20749 consensus_queue.cc:260] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:48.756855 20749 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.760814 20677 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:48.762161 20749 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:48.762219 21299 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:48.763375 21932 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:48.764070 21932 raft_consensus.cc:695] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:48.764989 21932 consensus_queue.cc:237] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:48.776276 21904 raft_consensus.cc:988] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:48.776878 21904 raft_consensus.cc:1076] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:48.776674 20505 catalog_manager.cc:5526] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:48.778867 20749 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:48.779479 20749 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:48.780637 20749 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.789011 20749 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:48.791064 20749 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:48.791779 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:48.792095 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:48.792498 20681 raft_consensus.cc:3049] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:48.792618 21335 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.792873 20681 raft_consensus.cc:738] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:48.793658 20681 consensus_queue.cc:260] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:48.795138 20681 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.801295 21335 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:48.802837 20720 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:48.803021 20681 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:48.803824 21810 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:48.804333 21810 raft_consensus.cc:695] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:48.805192 21810 consensus_queue.cc:237] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:48.827208 20505 catalog_manager.cc:5526] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:48.840008 21830 raft_consensus.cc:988] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:48.840553 21886 raft_consensus.cc:1076] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:48.842471 20677 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38583cae882144f48cec12c72ba697cf"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:34866
I20250116 02:58:48.843194 20677 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:48.843500 20677 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.849121 20677 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:48.850953 20677 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:48.851961 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:48.852108 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:48.853156 20752 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.859016 20603 raft_consensus.cc:3049] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:48.859426 20603 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:48.860271 20603 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:48.861524 20603 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:48.864091 20752 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:48.867007 20647 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:48.868999 21854 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:48.870362 21854 raft_consensus.cc:695] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:48.871387 21854 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:48.876807 20603 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:48.888911 20505 catalog_manager.cc:5526] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:49.045102 20752 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:49.046487 21928 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:49.064285 20681 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:49.069087 21886 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:49.074263 20677 consensus_queue.cc:237] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:49.082355 20605 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:49.084220 21904 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:49.090314 20752 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:49.092532 21904 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250116 02:58:49.130787 20646 consensus_peers.cc:487] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b1b1a15f38e4ebe9a2943597f827019. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:49.159677 20605 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:49.159724 20749 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:49.161636 21950 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:49.163022 21951 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:49.170791 21950 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:49.172601 20749 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:49.176303 20605 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:49.189715 20554 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20250116 02:58:49.191792 20504 catalog_manager.cc:5526] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:49.212263 20554 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 38583cae882144f48cec12c72ba697cf, TS f65e8d7f76144fab9725ca32a6328c3f -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:49.212641 20554 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 38583cae882144f48cec12c72ba697cf, TS f65e8d7f76144fab9725ca32a6328c3f -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:49.223083 20554 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 31db17a68ff141f3bd799ccc59c81fcc, TS 231d91433a404468a44ea0a107345aeb -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:49.223480 20554 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 31db17a68ff141f3bd799ccc59c81fcc, TS 231d91433a404468a44ea0a107345aeb -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:49.253458 20554 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7c32cae59d8843b993750c4fe4e543e0, TS f65e8d7f76144fab9725ca32a6328c3f -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:58:49.253876 20554 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7c32cae59d8843b993750c4fe4e543e0, TS f65e8d7f76144fab9725ca32a6328c3f -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:49.266425 20752 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:49.267867 21932 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:49.284327 20681 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:49.287565 21932 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:49.360534 20605 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:49.361994 21951 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:49.369606 21334 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:49.371562 21784 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:49.386107 20752 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:49.387846 21904 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:49.413990 20678 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:49.417006 21811 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:49.602593 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:49.683643 21966 ts_tablet_manager.cc:927] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:49.763002 21966 tablet_copy_client.cc:323] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:49.773355 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 5b1b1a15f38e4ebe9a2943597f827019 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:49.774291 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 5b1b1a15f38e4ebe9a2943597f827019 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-5b1b1a15f38e4ebe9a2943597f827019
I20250116 02:58:49.793596 20693 tablet_copy_source_session.cc:215] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:49.803507 21966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1b1a15f38e4ebe9a2943597f827019. 1 dirs total, 0 dirs full, 0 dirs failed
W20250116 02:58:49.812075 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 90400.5us
I20250116 02:58:49.846040 20678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:49.847240 20678 raft_consensus.cc:604] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:49.851179 20605 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:49.851962 20605 raft_consensus.cc:604] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 231d91433a404468a44ea0a107345aeb
I20250116 02:58:49.853629 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250116 02:58:49.855731 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:49.856559 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:49.874565 21966 tablet_copy_client.cc:806] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:49.877135 21966 tablet_copy_client.cc:670] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:49.885064 21966 tablet_copy_client.cc:538] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:49.896809 21966 tablet_bootstrap.cc:492] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:49.933857 21966 tablet_bootstrap.cc:492] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:49.934988 21966 tablet_bootstrap.cc:492] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:49.935837 21966 ts_tablet_manager.cc:1397] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.039s user 0.022s sys 0.012s
I20250116 02:58:49.939311 21966 raft_consensus.cc:357] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:49.940817 21966 raft_consensus.cc:738] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:49.941625 21966 consensus_queue.cc:260] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:49.944741 21966 ts_tablet_manager.cc:1428] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20250116 02:58:49.946837 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-5b1b1a15f38e4ebe9a2943597f827019 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:49.947290 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-5b1b1a15f38e4ebe9a2943597f827019 on tablet 5b1b1a15f38e4ebe9a2943597f827019 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:49.966847 21950 raft_consensus.cc:988] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:49.967439 21950 raft_consensus.cc:1076] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:49.970120 20752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:49.970880 20752 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:49.971915 20752 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:49.982432 20752 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:49.985124 20752 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:49.989818 21968 raft_consensus.cc:988] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:49.990439 21968 raft_consensus.cc:1076] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:49.992174 20678 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:34866
I20250116 02:58:49.992866 20678 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:49.993327 20678 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:49.993445 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:49.994282 21336 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:49.999784 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:50.000756 20681 raft_consensus.cc:3049] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:50.001261 20681 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:50.002450 20681 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:50.004521 20681 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:50.007395 21336 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:50.009178 20720 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:50.010205 21811 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:50.010576 21811 raft_consensus.cc:695] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:50.011399 21811 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:50.012951 20681 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:50.016438 20678 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:50.019806 20678 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:50.020833 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:50.021530 20752 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:50.023185 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:50.024112 20603 raft_consensus.cc:3049] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:50.024330 20603 raft_consensus.cc:738] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:50.025132 20603 consensus_queue.cc:260] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:50.026566 20603 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:50.030521 20752 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:50.032537 20647 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:50.033821 21854 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:50.034364 21854 raft_consensus.cc:695] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:50.038300 21854 consensus_queue.cc:237] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:50.049515 20504 catalog_manager.cc:5526] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:50.058310 20603 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:50.062099 20505 catalog_manager.cc:5526] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:50.246269 21334 raft_consensus.cc:1212] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:50.421856 20749 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:50.422919 21854 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:50.457671 20603 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:50.462378 21904 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:50.521581 20682 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:50.523077 21784 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:50.527168 21334 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:50.528730 21784 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:50.687716 21854 raft_consensus.cc:1059] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:50.690390 21854 consensus_queue.cc:237] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:50.700124 20752 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:50.700124 21336 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:50.701989 21950 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:50.702968 20603 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:50.704821 21904 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:50.706075 21950 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:50.724032 21854 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:50.727638 21336 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:50.728775 20603 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:50.746778 20752 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:50.755685 20505 catalog_manager.cc:5526] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:50.804059 20682 consensus_queue.cc:237] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:50.819200 20605 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:50.819433 21336 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:50.822103 21950 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:50.823261 21904 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:50.831357 21951 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:50.834255 20605 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:50.835569 21336 raft_consensus.cc:2949] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:50.845149 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5b1b1a15f38e4ebe9a2943597f827019 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:50.858147 20503 catalog_manager.cc:5526] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 3 to 4, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:50.881706 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1b1a15f38e4ebe9a2943597f827019 with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:50.899628 21994 tablet_replica.cc:331] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:50.900936 21994 raft_consensus.cc:2238] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:50.901841 21994 raft_consensus.cc:2267] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:50.906163 21994 ts_tablet_manager.cc:1905] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:50.924106 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:50.924540 21994 ts_tablet_manager.cc:1918] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:50.925053 21994 log.cc:1198] T 5b1b1a15f38e4ebe9a2943597f827019 P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/5b1b1a15f38e4ebe9a2943597f827019
I20250116 02:58:50.927726 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet 5b1b1a15f38e4ebe9a2943597f827019 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:50.968242 20752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:50.968990 20752 raft_consensus.cc:604] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:50.974416 20603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:50.975245 20603 raft_consensus.cc:604] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 231d91433a404468a44ea0a107345aeb
I20250116 02:58:50.981222 20682 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:50.981818 20682 raft_consensus.cc:604] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:50.984717 20752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:50.985334 20752 raft_consensus.cc:604] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:50.986799 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250116 02:58:50.988143 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:50.988706 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:51.286260 21983 raft_consensus.cc:988] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:51.286911 21965 raft_consensus.cc:1076] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:51.288534 20603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43850
I20250116 02:58:51.289124 20603 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:51.289398 20603 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:51.295285 20603 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:51.297916 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:51.298471 20678 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:51.298666 20603 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:51.300233 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:51.300875 20752 raft_consensus.cc:3049] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:51.301148 20752 raft_consensus.cc:738] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:51.301990 20752 consensus_queue.cc:260] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:51.303239 20752 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:51.306210 20678 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:51.307783 20571 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:51.308576 21963 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:51.309113 21963 raft_consensus.cc:695] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:51.309854 20752 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:51.310122 21963 consensus_queue.cc:237] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:51.321682 20503 catalog_manager.cc:5526] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:51.328084 21951 raft_consensus.cc:988] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:51.328634 21950 raft_consensus.cc:1076] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:51.330753 20752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:51.331655 20752 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:51.332077 20752 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:51.340531 20752 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:51.341984 20752 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:51.342912 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:51.343094 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:51.343530 21334 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:51.343583 20682 raft_consensus.cc:3049] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:51.344178 20682 raft_consensus.cc:738] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:51.344915 20682 consensus_queue.cc:260] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:51.346019 20682 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:51.350806 21334 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:51.351965 20720 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:51.353521 21965 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:51.354112 21965 raft_consensus.cc:695] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:51.356699 21965 consensus_queue.cc:237] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:51.357275 20682 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:51.367789 20503 catalog_manager.cc:5526] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:51.442816 21983 raft_consensus.cc:988] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:51.443418 21965 raft_consensus.cc:1076] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:51.444864 21334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56674
I20250116 02:58:51.445387 21334 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:51.445645 21334 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:51.452881 21334 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:51.455852 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:51.455845 21334 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:51.456538 20682 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:51.456913 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:51.457432 20752 raft_consensus.cc:3049] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:58:51.457707 20752 raft_consensus.cc:738] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:51.458377 20752 consensus_queue.cc:260] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:51.459749 20752 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:51.463217 20682 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:58:51.464428 21299 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:51.465354 21816 raft_consensus.cc:2798] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:58:51.465715 20752 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:58:51.466507 21816 raft_consensus.cc:695] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:51.467468 21816 consensus_queue.cc:237] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:51.476840 20503 catalog_manager.cc:5526] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 2 to 3, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 3 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:51.500939 21963 raft_consensus.cc:988] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:51.501473 21968 raft_consensus.cc:1076] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:51.503314 20682 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:34866
I20250116 02:58:51.503865 20682 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:51.504168 20682 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:51.510721 20682 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:51.513525 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:51.514007 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:51.514240 20603 raft_consensus.cc:3049] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:51.514636 21334 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:51.514554 20603 raft_consensus.cc:738] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:51.516351 20603 consensus_queue.cc:260] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:51.517557 20603 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:51.518736 20682 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:51.522909 21334 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:51.524446 20646 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:51.525132 20603 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:51.525245 21950 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:51.525806 21950 raft_consensus.cc:695] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:51.526767 21950 consensus_queue.cc:237] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:51.538255 20503 catalog_manager.cc:5526] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:51.768606 20749 raft_consensus.cc:1270] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:51.770002 21968 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:51.786415 20678 raft_consensus.cc:1270] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:51.789543 22003 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:51.817610 21334 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:51.821851 21965 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:51.850940 20678 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:51.856238 21965 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:51.956419 20605 consensus_queue.cc:237] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:51.965596 20752 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:51.967697 20682 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250116 02:58:51.969858 20571 consensus_peers.cc:487] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9815466261546de9d937eeb8cd5cd2e. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:51.970592 22003 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:51.971688 21963 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:51.981618 22003 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:51.983624 20678 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:51.986208 20749 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:58:51.985960 20752 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:51.988166 21997 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250116 02:58:51.998852 20682 consensus_queue.cc:237] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:52.004267 20503 catalog_manager.cc:5526] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:52.031114 20646 consensus_peers.cc:487] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31f82fb4c6a24889be1c1e2fe3e7d3e5. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:52.031356 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:52.042995 20757 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250116 02:58:52.045871 20605 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250116 02:58:52.046123 21951 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:52.048405 21951 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:52.067684 21332 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:52.069512 20680 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:58:52.079069 21950 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:52.100566 21950 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.143766 20601 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:52.146983 20752 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.161893 22019 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:52.178886 20752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:52.179632 20752 raft_consensus.cc:604] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:52.184454 21816 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:58:52.192414 20605 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:52.201457 20605 raft_consensus.cc:604] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 231d91433a404468a44ea0a107345aeb
W20250116 02:58:52.203797 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 57738.8us
I20250116 02:58:52.198246 20752 consensus_queue.cc:237] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:52.216907 20682 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:52.219915 20682 raft_consensus.cc:604] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:52.250497 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250116 02:58:52.253041 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:52.253917 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:52.254465 20503 catalog_manager.cc:5526] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:52.280280 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 45542.5us
W20250116 02:58:52.295009 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 63411.6us
I20250116 02:58:52.337020 20603 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
W20250116 02:58:52.339314 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 85251.6us
I20250116 02:58:52.386947 20682 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250116 02:58:52.408026 21950 raft_consensus.cc:988] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:52.414253 22024 raft_consensus.cc:1076] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
W20250116 02:58:52.434849 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 89698.4us
W20250116 02:58:52.444244 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 64863us
W20250116 02:58:52.448544 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 56379.3us
I20250116 02:58:52.453892 20608 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250116 02:58:52.455825 22036 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:52.461994 22032 raft_consensus.cc:988] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:52.462596 22032 raft_consensus.cc:1076] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 231d91433a404468a44ea0a107345aeb to start an election
W20250116 02:58:52.467911 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 71843.8us
I20250116 02:58:52.480103 20680 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:34866
I20250116 02:58:52.480857 20680 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:52.481244 20680 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:52.490581 22036 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.501255 20605 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.514580 22050 ts_tablet_manager.cc:927] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:52.522998 20752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:52.545210 20752 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:52.545702 20752 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:52.542080 20680 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:52.553623 20680 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:52.555083 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:52.555841 20607 raft_consensus.cc:3049] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:52.555936 22052 ts_tablet_manager.cc:927] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:52.556216 20607 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:52.557190 20607 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:52.559248 20607 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:52.573103 20752 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:52.551700 22050 tablet_copy_client.cc:323] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:52.576261 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet d9815466261546de9d937eeb8cd5cd2e from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:52.576974 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet d9815466261546de9d937eeb8cd5cd2e from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-d9815466261546de9d937eeb8cd5cd2e
I20250116 02:58:52.580907 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:52.581473 20752 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:52.582161 20608 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:58:52.584343 20720 consensus_peers.cc:487] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d3fee7b273345558c661eda0710029c. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:52.595188 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:52.596316 20503 catalog_manager.cc:5526] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.599656 20680 consensus_queue.cc:237] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:52.605629 20607 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:52.610066 20649 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:52.610522 20608 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:52.609606 20618 tablet_copy_source_session.cc:215] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:52.613070 20723 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:52.620071 22028 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:52.620262 21951 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:52.634275 21951 raft_consensus.cc:695] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:52.635180 22050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9815466261546de9d937eeb8cd5cd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:52.625905 22052 tablet_copy_client.cc:323] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:52.642416 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:52.643030 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-31f82fb4c6a24889be1c1e2fe3e7d3e5
I20250116 02:58:52.646703 21965 raft_consensus.cc:988] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:52.648001 21965 raft_consensus.cc:1076] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:52.631357 22036 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:52.635672 21951 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:52.600546 21332 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:52.638305 22028 raft_consensus.cc:695] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:52.652118 22028 consensus_queue.cc:237] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:52.658593 20607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43850
I20250116 02:58:52.692157 20607 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:52.692659 20607 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:58:52.692787 20646 consensus_peers.cc:487] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3ccbf96c6c547e19daa8f5424fa43ee. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:52.694144 20605 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:52.667735 21332 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:52.687322 20505 catalog_manager.cc:5526] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:52.696570 21951 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:52.686604 22050 tablet_copy_client.cc:806] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:52.697832 22050 tablet_copy_client.cc:670] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:52.669627 20693 tablet_copy_source_session.cc:215] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:52.687467 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:52.701902 20682 raft_consensus.cc:3049] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:52.702214 20682 raft_consensus.cc:738] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:52.703342 20607 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:52.703786 22052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:52.703912 20682 consensus_queue.cc:260] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:52.705139 20682 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:52.713425 22050 tablet_copy_client.cc:538] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:52.717677 20607 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:52.717829 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:52.718426 20678 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:52.724325 20503 catalog_manager.cc:5526] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:52.726904 22052 tablet_copy_client.cc:806] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:52.725281 20678 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:58:52.728509 20571 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:52.729050 22052 tablet_copy_client.cc:670] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:52.729395 21963 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:58:52.730343 21963 raft_consensus.cc:695] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:52.731434 21963 consensus_queue.cc:237] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:52.732964 20756 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:58:52.733845 20682 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:52.736444 22052 tablet_copy_client.cc:538] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:52.742554 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:52.743388 20752 raft_consensus.cc:3049] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:58:52.743687 20752 raft_consensus.cc:738] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:52.743872 20749 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:58:52.744540 20752 consensus_queue.cc:260] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:52.745990 20752 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:52.766184 22052 tablet_bootstrap.cc:492] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:52.782446 22050 tablet_bootstrap.cc:492] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:52.783744 20505 catalog_manager.cc:5526] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 2 to 3, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 3 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:52.785868 20752 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:58:52.787425 22027 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:52.793315 20682 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.802132 21951 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.808610 20605 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.859385 22063 ts_tablet_manager.cc:927] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:52.865392 20505 catalog_manager.cc:5526] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.868503 22063 tablet_copy_client.cc:323] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.131:43647
I20250116 02:58:52.865481 22052 tablet_bootstrap.cc:492] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:52.871771 22052 tablet_bootstrap.cc:492] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:52.872794 22052 ts_tablet_manager.cc:1397] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.107s user 0.032s sys 0.009s
I20250116 02:58:52.874285 20766 tablet_copy_service.cc:140] P 17446c8dacf04461acb843547665250f: Received BeginTabletCopySession request for tablet 5d3fee7b273345558c661eda0710029c from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:35540)
I20250116 02:58:52.875118 20766 tablet_copy_service.cc:161] P 17446c8dacf04461acb843547665250f: Beginning new tablet copy session on tablet 5d3fee7b273345558c661eda0710029c from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:35540: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-5d3fee7b273345558c661eda0710029c
I20250116 02:58:52.893777 20766 tablet_copy_source_session.cc:215] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:52.893584 20756 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:52.906798 22063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d3fee7b273345558c661eda0710029c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:52.911309 22052 raft_consensus.cc:357] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:52.925642 22052 raft_consensus.cc:738] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:52.926481 22052 consensus_queue.cc:260] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:52.933610 22052 ts_tablet_manager.cc:1428] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.060s user 0.007s sys 0.000s
I20250116 02:58:52.939625 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-31f82fb4c6a24889be1c1e2fe3e7d3e5 received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:52.940184 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-31f82fb4c6a24889be1c1e2fe3e7d3e5 on tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:52.942214 22050 tablet_bootstrap.cc:492] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:52.943382 22050 tablet_bootstrap.cc:492] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:52.952127 22050 ts_tablet_manager.cc:1397] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.170s user 0.033s sys 0.039s
I20250116 02:58:52.961505 22050 raft_consensus.cc:357] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:52.962138 22063 tablet_copy_client.cc:806] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:52.962623 22050 raft_consensus.cc:738] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:52.963207 22063 tablet_copy_client.cc:670] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:52.964000 22050 consensus_queue.cc:260] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:52.969170 22050 ts_tablet_manager.cc:1428] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20250116 02:58:52.968096 22063 tablet_copy_client.cc:538] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:52.976552 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-d9815466261546de9d937eeb8cd5cd2e received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:52.977046 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-d9815466261546de9d937eeb8cd5cd2e on tablet d9815466261546de9d937eeb8cd5cd2e with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:52.981434 22063 tablet_bootstrap.cc:492] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:53.013166 21332 raft_consensus.cc:1212] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250116 02:58:53.023942 22063 tablet_bootstrap.cc:492] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:53.025315 22063 tablet_bootstrap.cc:492] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:53.027086 22063 ts_tablet_manager.cc:1397] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.015s
I20250116 02:58:53.030277 22063 raft_consensus.cc:357] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:53.031086 22063 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:53.032972 22063 consensus_queue.cc:260] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:53.037057 22063 ts_tablet_manager.cc:1428] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250116 02:58:53.039358 20766 tablet_copy_service.cc:342] P 17446c8dacf04461acb843547665250f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-5d3fee7b273345558c661eda0710029c received from {username='slave'} at 127.0.0.1:35540
I20250116 02:58:53.039896 20766 tablet_copy_service.cc:434] P 17446c8dacf04461acb843547665250f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-5d3fee7b273345558c661eda0710029c on tablet 5d3fee7b273345558c661eda0710029c with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:53.067088 20607 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:53.069433 22019 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:53.090265 20608 raft_consensus.cc:1270] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:53.092612 22028 consensus_queue.cc:1035] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:53.093808 20682 raft_consensus.cc:1270] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:53.096004 22035 consensus_queue.cc:1035] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:53.137823 21332 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:53.145402 22019 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:53.163286 20678 raft_consensus.cc:1270] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:53.180850 21963 consensus_queue.cc:1035] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
W20250116 02:58:53.207571 22070 log.cc:926] Time spent T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250116 02:58:53.217995 21334 raft_consensus.cc:1212] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250116 02:58:53.254901 22063 ts_tablet_manager.cc:927] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:53.265395 20757 raft_consensus.cc:1270] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:53.267617 21332 raft_consensus.cc:1212] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:53.279883 22003 consensus_queue.cc:1035] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:53.285753 22063 tablet_copy_client.cc:323] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:53.293560 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet a3ccbf96c6c547e19daa8f5424fa43ee from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:53.294255 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet a3ccbf96c6c547e19daa8f5424fa43ee from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-a3ccbf96c6c547e19daa8f5424fa43ee
I20250116 02:58:53.310526 20693 tablet_copy_source_session.cc:215] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:53.314955 22063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ccbf96c6c547e19daa8f5424fa43ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:53.317993 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:53.649945 22063 tablet_copy_client.cc:806] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:53.650718 22063 tablet_copy_client.cc:670] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
W20250116 02:58:53.652204 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 52051.8us
I20250116 02:58:53.667419 20749 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:53.668239 20749 raft_consensus.cc:604] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:53.780557 20608 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:53.781337 20608 raft_consensus.cc:604] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
W20250116 02:58:53.807551 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 103924us
W20250116 02:58:53.818833 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 111968us
I20250116 02:58:53.832597 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:53.833424 20681 raft_consensus.cc:604] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:53.849406 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250116 02:58:53.856900 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:53.857759 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:53.865186 22082 raft_consensus.cc:988] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:53.865932 22082 raft_consensus.cc:1076] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:53.866796 22094 raft_consensus.cc:1059] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:53.869956 22094 consensus_queue.cc:237] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:53.878144 20756 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:53.879078 20756 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:53.879439 20756 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:53.891754 20756 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:53.894093 20756 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:53.895967 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:53.896719 20682 raft_consensus.cc:3049] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:58:53.897020 20682 raft_consensus.cc:738] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:53.897769 20682 consensus_queue.cc:260] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:53.899796 20682 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:53.900509 20756 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:53.902820 22092 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:53.908582 20682 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:58:53.909965 20720 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:53.910946 22085 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:58:53.911401 22085 raft_consensus.cc:695] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
W20250116 02:58:53.910948 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 62897.7us
I20250116 02:58:53.912426 22085 consensus_queue.cc:237] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:53.918583 20603 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20250116 02:58:53.912341 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 58786.2us
I20250116 02:58:53.928998 22063 tablet_copy_client.cc:538] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:53.930740 22028 raft_consensus.cc:1059] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:53.933406 22028 consensus_queue.cc:237] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:53.935250 21335 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:53.937043 22094 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:53.942472 22063 tablet_bootstrap.cc:492] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:53.944027 22081 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:53.950718 20505 catalog_manager.cc:5526] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 3, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 3 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:53.960583 20677 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:53.962240 21983 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250116 02:58:53.968230 21335 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:53.982306 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:53.983255 20607 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:53.984432 22085 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:53.985275 22003 raft_consensus.cc:1059] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:53.996227 22081 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:53.998195 22075 raft_consensus.cc:988] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:54.003225 22075 raft_consensus.cc:1076] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:54.003619 22003 consensus_queue.cc:237] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:54.005175 20607 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:58:54.008303 20603 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:54.014955 21335 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.023172 20756 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:54.027561 22026 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:54.037294 20752 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.059302 20603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43850
I20250116 02:58:54.059996 20603 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:54.060396 20603 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:54.071105 22026 raft_consensus.cc:988] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:54.071658 22026 raft_consensus.cc:1076] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:54.075855 21335 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:54.077243 20678 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:54.070250 20503 catalog_manager.cc:5526] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.080497 20603 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:54.092239 20607 raft_consensus.cc:1235] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 1. Current term is 2. Ops: []
I20250116 02:58:54.097113 20601 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.098814 22003 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:54.102133 22087 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:54.107080 22087 consensus_queue.cc:1046] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:58:54.108353 22028 raft_consensus.cc:3049] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:54.108742 22028 raft_consensus.cc:738] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:54.109678 22028 consensus_queue.cc:260] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:54.115231 22028 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:54.119169 22003 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:54.127488 20603 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:54.131672 22063 tablet_bootstrap.cc:492] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:54.134387 22063 tablet_bootstrap.cc:492] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:54.137969 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:54.140141 22063 ts_tablet_manager.cc:1397] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.198s user 0.023s sys 0.022s
W20250116 02:58:54.140583 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 49837.5us
I20250116 02:58:54.148427 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:54.149137 20678 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:54.168228 20756 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50646
I20250116 02:58:54.169904 20756 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:54.171106 20756 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:54.176132 20678 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:54.183882 20571 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:54.232470 20756 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:54.237766 20756 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:54.241475 22063 raft_consensus.cc:357] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:54.232599 22026 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election won for term 2
W20250116 02:58:54.279809 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 96021.5us
W20250116 02:58:54.279590 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 99528.2us
I20250116 02:58:54.263895 22063 raft_consensus.cc:738] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:54.287371 22063 consensus_queue.cc:260] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:54.248283 22090 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.269856 22026 raft_consensus.cc:695] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:54.295454 22026 consensus_queue.cc:237] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:54.312944 20752 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:54.314775 20680 consensus_queue.cc:237] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:54.323740 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:54.325615 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:54.377125 20607 raft_consensus.cc:3049] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:54.377722 20607 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:54.387720 20607 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:58:54.389909 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 55875.2us
I20250116 02:58:54.399322 20607 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:58:54.403936 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 78790.5us
I20250116 02:58:54.365099 21332 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:58:54.383256 22097 log.cc:926] Time spent T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20250116 02:58:54.426810 22063 ts_tablet_manager.cc:1428] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.285s user 0.008s sys 0.002s
W20250116 02:58:54.430220 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 87497.1us
I20250116 02:58:54.416975 20607 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
W20250116 02:58:54.421224 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 55639.6us
I20250116 02:58:54.496851 21332 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:54.518898 22026 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.522179 20607 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:54.523296 20505 catalog_manager.cc:5526] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:54.531345 21332 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.543147 20682 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.563306 21335 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:54.572660 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-a3ccbf96c6c547e19daa8f5424fa43ee received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:54.573274 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-a3ccbf96c6c547e19daa8f5424fa43ee on tablet a3ccbf96c6c547e19daa8f5424fa43ee with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:54.576915 22136 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:54.580770 21333 raft_consensus.cc:1212] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250116 02:58:54.583583 20723 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:54.584254 20608 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.585971 22126 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:54.586521 22126 raft_consensus.cc:695] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:54.587549 22126 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:54.593622 22136 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 02:58:54.585992 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 208306us
I20250116 02:58:54.601239 21333 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.588187 22075 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250116 02:58:54.626055 20682 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:54.626881 20677 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.617908 20752 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:54.620056 21334 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.676753 22137 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:54.682267 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:54.681795 20503 catalog_manager.cc:5526] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.691121 20505 catalog_manager.cc:5526] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:54.711439 22124 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:54.729701 20756 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.730638 20753 consensus_queue.cc:237] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:54.743008 20752 raft_consensus.cc:2949] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.746690 20504 catalog_manager.cc:5526] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 4 to 5, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:54.748275 20601 consensus_queue.cc:237] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:54.757787 21334 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:54.787585 20601 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:54.794175 22126 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:54.798343 21334 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:54.800181 22093 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:54.801738 22085 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:54.811581 22126 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.828648 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5 with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:54.830273 21333 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.835247 20601 raft_consensus.cc:2949] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.839218 20491 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5d3fee7b273345558c661eda0710029c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:54.851577 20752 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:54.840590 22003 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.856647 22146 tablet_replica.cc:331] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:54.855732 22118 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20250116 02:58:54.858696 22146 raft_consensus.cc:2238] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:54.859645 22146 raft_consensus.cc:2267] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:54.844308 21332 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.868388 20677 raft_consensus.cc:1270] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:54.868530 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d9815466261546de9d937eeb8cd5cd2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:54.867748 20752 raft_consensus.cc:2949] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.844381 20505 catalog_manager.cc:5526] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:54.871228 22031 consensus_queue.cc:1035] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:54.876221 22146 ts_tablet_manager.cc:1905] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:54.879387 20503 catalog_manager.cc:5526] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 3 to 4, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 1 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:54.894045 22128 raft_consensus.cc:1059] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:54.897068 22128 consensus_queue.cc:237] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:58:54.911131 20491 catalog_manager.cc:5049] ChangeConfig:REMOVE_PEER RPC for tablet 5d3fee7b273345558c661eda0710029c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 17446c8dacf04461acb843547665250f (127.18.238.131:43647) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20250116 02:58:54.915643 20503 catalog_manager.cc:5526] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 4 to 5, VOTER 231d91433a404468a44ea0a107345aeb (127.18.238.130) evicted. New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:54.918275 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet d9815466261546de9d937eeb8cd5cd2e with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:54.929322 20756 raft_consensus.cc:1270] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:54.931893 22146 ts_tablet_manager.cc:1918] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250116 02:58:54.944126 22146 log.cc:1198] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/31f82fb4c6a24889be1c1e2fe3e7d3e5
I20250116 02:58:54.932119 22003 consensus_queue.cc:1035] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20250116 02:58:54.946584 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet 31f82fb4c6a24889be1c1e2fe3e7d3e5 (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:54.959128 20666 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3fee7b273345558c661eda0710029c with delete_type TABLET_DATA_TOMBSTONED (TS 231d91433a404468a44ea0a107345aeb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34864
I20250116 02:58:54.981542 20608 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:54.983557 22085 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:54.992580 22148 tablet_replica.cc:331] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:54.994968 21333 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:54.997750 20752 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:55.013792 22128 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:55.014897 22136 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:55.007334 20601 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250116 02:58:55.022163 22148 raft_consensus.cc:2238] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:55.031890 22148 raft_consensus.cc:2267] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:55.030337 22128 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:55.059707 22148 ts_tablet_manager.cc:1905] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:55.076140 22119 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:55.078063 20752 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:55.085371 20608 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:55.107007 20505 catalog_manager.cc:5526] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:55.167855 21333 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:55.217767 20683 consensus_queue.cc:237] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:55.223202 22148 ts_tablet_manager.cc:1918] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:55.223732 22148 log.cc:1198] T d9815466261546de9d937eeb8cd5cd2e P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/d9815466261546de9d937eeb8cd5cd2e
I20250116 02:58:55.225368 22148 tablet_replica.cc:331] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: stopping tablet replica
I20250116 02:58:55.225643 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet d9815466261546de9d937eeb8cd5cd2e (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
W20250116 02:58:55.230342 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 78383us
I20250116 02:58:55.236294 21334 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:55.238900 21332 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:55.240480 22129 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:55.248957 20753 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250116 02:58:55.250706 22156 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:55.250398 22092 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:55.254577 22131 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:55.264180 20753 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:55.268141 22148 raft_consensus.cc:2238] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:55.269429 22148 raft_consensus.cc:2267] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:55.268718 21332 raft_consensus.cc:2949] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:55.277244 22148 ts_tablet_manager.cc:1905] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:55.279618 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a3ccbf96c6c547e19daa8f5424fa43ee with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:55.297305 20502 catalog_manager.cc:5526] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f reported cstate change: config changed from index 3 to 4, VOTER f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) evicted. New cstate: current_term: 1 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:55.306597 22148 ts_tablet_manager.cc:1918] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250116 02:58:55.307237 22148 log.cc:1198] T 5d3fee7b273345558c661eda0710029c P 231d91433a404468a44ea0a107345aeb: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-1-root/wals/5d3fee7b273345558c661eda0710029c
I20250116 02:58:55.308884 20490 catalog_manager.cc:4872] TS 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): tablet 5d3fee7b273345558c661eda0710029c (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:55.316354 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ccbf96c6c547e19daa8f5424fa43ee with delete_type TABLET_DATA_TOMBSTONED (TS f65e8d7f76144fab9725ca32a6328c3f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43834
I20250116 02:58:55.317395 22146 tablet_replica.cc:331] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f: stopping tablet replica
I20250116 02:58:55.318456 22146 raft_consensus.cc:2238] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:55.319283 22146 raft_consensus.cc:2267] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:55.322813 22146 ts_tablet_manager.cc:1905] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:55.339356 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:55.343963 22146 ts_tablet_manager.cc:1918] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250116 02:58:55.344386 22146 log.cc:1198] T a3ccbf96c6c547e19daa8f5424fa43ee P f65e8d7f76144fab9725ca32a6328c3f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-0-root/wals/a3ccbf96c6c547e19daa8f5424fa43ee
I20250116 02:58:55.345803 20493 catalog_manager.cc:4872] TS f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): tablet a3ccbf96c6c547e19daa8f5424fa43ee (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
I20250116 02:58:55.383224 20753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:55.383881 20753 raft_consensus.cc:604] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:55.387066 20677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:55.387719 20677 raft_consensus.cc:604] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:55.391790 20753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:58:55.392356 20753 raft_consensus.cc:604] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:55.395114 20608 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:55.395596 20608 raft_consensus.cc:604] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Received request to transfer leadership to TS 231d91433a404468a44ea0a107345aeb
I20250116 02:58:55.396976 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250116 02:58:55.398131 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:55.398808 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:55.616495 22119 raft_consensus.cc:988] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:55.617136 22119 raft_consensus.cc:1076] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:55.618956 20753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:55.619518 20753 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:55.620076 20753 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:55.626243 20753 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:55.627964 20753 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:55.629946 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:55.630579 20606 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:55.631721 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:55.632385 20677 raft_consensus.cc:3049] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:55.632794 20677 raft_consensus.cc:738] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:55.639164 20606 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:55.640264 22155 raft_consensus.cc:988] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:55.640182 20677 consensus_queue.cc:260] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:55.640672 20723 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:55.640884 22117 raft_consensus.cc:1076] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:55.641741 20677 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:55.641852 22155 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:55.642362 22155 raft_consensus.cc:695] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:55.643311 22155 consensus_queue.cc:237] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:55.643853 20606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43850
I20250116 02:58:55.644421 20606 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:55.644781 20606 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:55.652802 22126 raft_consensus.cc:988] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:55.653439 22126 raft_consensus.cc:1076] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:58:55.657895 21333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56674
I20250116 02:58:55.658495 21333 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:55.658881 21333 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:55.652657 20606 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:55.664291 20606 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:55.668823 20677 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:55.671191 21333 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:55.675199 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:55.675935 20683 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:55.677274 20502 catalog_manager.cc:5526] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:58:55.682260 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:55.682658 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:55.683027 20749 raft_consensus.cc:3049] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:55.683367 20749 raft_consensus.cc:738] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:55.684145 20749 consensus_queue.cc:260] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:55.685428 20749 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:55.688503 21333 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:55.690456 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:55.691298 20753 raft_consensus.cc:3049] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:55.691673 20753 raft_consensus.cc:738] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:55.692396 20683 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:55.692768 20606 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:55.692668 20753 consensus_queue.cc:260] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:55.693671 20571 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:55.694317 20753 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:55.694495 22118 raft_consensus.cc:2798] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:55.695045 22118 raft_consensus.cc:695] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:55.695986 22118 consensus_queue.cc:237] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:55.707509 20753 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:55.707509 20606 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:58:55.708843 21300 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:55.711339 22043 raft_consensus.cc:2798] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:55.711453 20749 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:55.711782 22043 raft_consensus.cc:695] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:58:55.713146 22043 consensus_queue.cc:237] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:55.722553 20502 catalog_manager.cc:5526] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:55.726544 20503 catalog_manager.cc:5526] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:58:55.969009 22157 raft_consensus.cc:988] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:55.970259 22157 raft_consensus.cc:1076] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Signalling peer 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:55.971652 20683 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:34866
I20250116 02:58:55.972187 20683 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:55.972560 20683 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:55.978152 20683 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:55.979738 20683 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:55.980551 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:55.980667 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:55.981444 20606 raft_consensus.cc:3049] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:58:55.981518 20749 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:55.981863 20606 raft_consensus.cc:738] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:55.982890 20606 consensus_queue.cc:260] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:55.984086 20606 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:55.989025 20749 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:58:55.990201 20647 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:58:55.991098 22092 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:58:55.991032 20606 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:58:55.992230 22092 raft_consensus.cc:695] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:55.993180 22092 consensus_queue.cc:237] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:56.003346 20503 catalog_manager.cc:5526] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:56.055531 20606 raft_consensus.cc:1270] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:56.059154 22117 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:56.074715 20683 raft_consensus.cc:1270] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:56.081351 22117 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:56.132304 20753 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:56.133896 22163 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:56.137023 20683 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:58:56.138716 22118 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:58:56.196106 20756 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:56.198063 22043 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:56.210851 20608 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:56.212545 22079 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:56.369671 20606 consensus_queue.cc:237] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:56.375851 20753 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250116 02:58:56.376227 20571 consensus_peers.cc:487] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2960f90503b44912b4b4b58cd2ceb89f. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:56.377110 20676 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250116 02:58:56.377228 22163 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:56.379062 22163 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:56.388932 22157 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.391729 20753 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.391798 20676 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.410450 20502 catalog_manager.cc:5526] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.414433 20676 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:56.426756 20606 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250116 02:58:56.427992 20753 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250116 02:58:56.429316 20646 consensus_peers.cc:487] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38583cae882144f48cec12c72ba697cf. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:56.429411 22119 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:56.431481 22092 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:56.440328 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:56.444787 22092 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.457463 20606 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.463073 20756 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.483141 20503 catalog_manager.cc:5526] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.484903 20753 consensus_queue.cc:237] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:56.515779 20676 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250116 02:58:56.516232 20720 consensus_peers.cc:487] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf7916e71bc94e1eba4ab8fd75257e48. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:56.516644 20608 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:56.517213 20608 raft_consensus.cc:604] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 231d91433a404468a44ea0a107345aeb
I20250116 02:58:56.521346 20608 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250116 02:58:56.521147 22126 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:56.524190 22126 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:56.525048 20753 raft_consensus.cc:1270] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:56.525516 20683 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:56.528554 20683 raft_consensus.cc:604] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:58:56.529456 22092 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:56.530500 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250116 02:58:56.533715 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:56.534447 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:56.543645 22117 raft_consensus.cc:2949] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.547369 20608 raft_consensus.cc:2949] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.552752 20676 raft_consensus.cc:2949] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.569463 20607 raft_consensus.cc:1270] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:58:56.569778 20606 consensus_queue.cc:237] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
W20250116 02:58:56.585662 20571 consensus_peers.cc:487] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8dd85faff294418bdf7bf8a44f21ea0. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:56.587388 22092 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:56.587206 20756 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250116 02:58:56.596050 20676 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250116 02:58:56.596511 22118 raft_consensus.cc:988] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:56.596963 22118 raft_consensus.cc:1076] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:56.599843 22157 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:58:56.601176 22163 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:58:56.608808 22092 raft_consensus.cc:988] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:56.611708 22119 raft_consensus.cc:1076] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:58:56.613309 20608 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43854
I20250116 02:58:56.613943 20608 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:56.613456 22118 raft_consensus.cc:2949] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.614828 20608 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:56.614892 20505 catalog_manager.cc:5526] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.629913 20683 raft_consensus.cc:2949] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.621284 20756 raft_consensus.cc:2949] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:58:56.601543 20676 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:34866
I20250116 02:58:56.647990 20676 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:56.648443 20676 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:56.654898 20608 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:56.658144 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:56.658836 20683 raft_consensus.cc:3049] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:56.659129 20683 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:56.659813 20683 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:56.660902 20683 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:56.666492 20608 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:56.667594 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:56.668370 21333 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:56.669551 20683 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:56.670821 20571 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:56.677451 21333 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:56.705888 20676 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:56.682441 22157 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:56.684660 20504 catalog_manager.cc:5526] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index -1 to 3, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250116 02:58:56.702132 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 43987.6us
I20250116 02:58:56.714565 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:56.715270 20606 raft_consensus.cc:3049] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:56.715545 20606 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:56.716218 20606 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:56.717268 20606 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:56.718045 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:58:56.718655 21333 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:56.720118 22157 raft_consensus.cc:695] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:56.721274 22157 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:56.717521 20676 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:56.727823 21333 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:56.729302 20646 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:56.730396 22119 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:56.730943 22119 raft_consensus.cc:695] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:58:56.714496 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 43751us
I20250116 02:58:56.731812 22119 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:56.740473 20606 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:56.763769 20505 catalog_manager.cc:5526] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:56.763528 20504 catalog_manager.cc:5526] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:56.781433 22194 ts_tablet_manager.cc:927] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:56.792112 22194 tablet_copy_client.cc:323] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:56.793816 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet 2960f90503b44912b4b4b58cd2ceb89f from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:56.794322 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet 2960f90503b44912b4b4b58cd2ceb89f from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-2960f90503b44912b4b4b58cd2ceb89f
I20250116 02:58:56.802274 20618 tablet_copy_source_session.cc:215] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:56.805717 22194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2960f90503b44912b4b4b58cd2ceb89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:56.822022 22194 tablet_copy_client.cc:806] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:56.822961 22194 tablet_copy_client.cc:670] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:56.828601 22194 tablet_copy_client.cc:538] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:56.841722 22196 ts_tablet_manager.cc:927] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:56.844870 22194 tablet_bootstrap.cc:492] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:56.850057 22196 tablet_copy_client.cc:323] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:58:56.851616 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 38583cae882144f48cec12c72ba697cf from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:58:56.852094 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 38583cae882144f48cec12c72ba697cf from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-38583cae882144f48cec12c72ba697cf
I20250116 02:58:56.861089 20693 tablet_copy_source_session.cc:215] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:56.864801 22196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38583cae882144f48cec12c72ba697cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:56.873083 22194 tablet_bootstrap.cc:492] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:56.874218 22194 tablet_bootstrap.cc:492] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:56.875056 22194 ts_tablet_manager.cc:1397] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.012s
I20250116 02:58:56.878232 22194 raft_consensus.cc:357] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:56.879276 22194 raft_consensus.cc:738] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:56.879917 22194 consensus_queue.cc:260] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:56.882572 22194 ts_tablet_manager.cc:1428] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250116 02:58:56.884593 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-2960f90503b44912b4b4b58cd2ceb89f received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:56.885087 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-2960f90503b44912b4b4b58cd2ceb89f on tablet 2960f90503b44912b4b4b58cd2ceb89f with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:56.886023 22196 tablet_copy_client.cc:806] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:56.886687 22196 tablet_copy_client.cc:670] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:56.896742 22196 tablet_copy_client.cc:538] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:56.907210 22196 tablet_bootstrap.cc:492] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:56.936638 22196 tablet_bootstrap.cc:492] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:56.937654 22196 tablet_bootstrap.cc:492] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:56.938390 22196 ts_tablet_manager.cc:1397] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.001s
I20250116 02:58:56.941645 22196 raft_consensus.cc:357] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:56.942601 22196 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:56.945767 22196 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:56.954115 22196 ts_tablet_manager.cc:1428] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250116 02:58:56.956023 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-38583cae882144f48cec12c72ba697cf received from {username='slave'} at 127.0.0.1:41212
I20250116 02:58:56.956507 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-38583cae882144f48cec12c72ba697cf on tablet 38583cae882144f48cec12c72ba697cf with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:56.984534 22196 ts_tablet_manager.cc:927] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:58:56.986294 22196 tablet_copy_client.cc:323] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.129:45195
I20250116 02:58:56.987823 20618 tablet_copy_service.cc:140] P f65e8d7f76144fab9725ca32a6328c3f: Received BeginTabletCopySession request for tablet e8dd85faff294418bdf7bf8a44f21ea0 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:53964)
I20250116 02:58:56.988380 20618 tablet_copy_service.cc:161] P f65e8d7f76144fab9725ca32a6328c3f: Beginning new tablet copy session on tablet e8dd85faff294418bdf7bf8a44f21ea0 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:53964: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-e8dd85faff294418bdf7bf8a44f21ea0
I20250116 02:58:56.995930 20618 tablet_copy_source_session.cc:215] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:56.999092 22196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8dd85faff294418bdf7bf8a44f21ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:57.014191 22196 tablet_copy_client.cc:806] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:57.014880 22196 tablet_copy_client.cc:670] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:57.024344 22196 tablet_copy_client.cc:538] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:57.033488 22196 tablet_bootstrap.cc:492] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:57.055305 22196 tablet_bootstrap.cc:492] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:57.056121 22196 tablet_bootstrap.cc:492] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:57.056737 22196 ts_tablet_manager.cc:1397] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.003s
I20250116 02:58:57.059551 22196 raft_consensus.cc:357] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:57.061803 22196 raft_consensus.cc:738] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:57.062469 22196 consensus_queue.cc:260] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:57.065670 22196 ts_tablet_manager.cc:1428] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20250116 02:58:57.068682 20618 tablet_copy_service.cc:342] P f65e8d7f76144fab9725ca32a6328c3f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-e8dd85faff294418bdf7bf8a44f21ea0 received from {username='slave'} at 127.0.0.1:53964
I20250116 02:58:57.069159 20618 tablet_copy_service.cc:434] P f65e8d7f76144fab9725ca32a6328c3f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-e8dd85faff294418bdf7bf8a44f21ea0 on tablet e8dd85faff294418bdf7bf8a44f21ea0 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:57.093500 22196 ts_tablet_manager.cc:927] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:57.097719 22196 tablet_copy_client.cc:323] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.131:43647
I20250116 02:58:57.099435 20766 tablet_copy_service.cc:140] P 17446c8dacf04461acb843547665250f: Received BeginTabletCopySession request for tablet bf7916e71bc94e1eba4ab8fd75257e48 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:35540)
I20250116 02:58:57.100052 20766 tablet_copy_service.cc:161] P 17446c8dacf04461acb843547665250f: Beginning new tablet copy session on tablet bf7916e71bc94e1eba4ab8fd75257e48 from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:35540: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-bf7916e71bc94e1eba4ab8fd75257e48
I20250116 02:58:57.105911 20766 tablet_copy_source_session.cc:215] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:58:57.109127 22196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7916e71bc94e1eba4ab8fd75257e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:58:57.128268 22196 tablet_copy_client.cc:806] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
I20250116 02:58:57.129148 22196 tablet_copy_client.cc:670] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:58:57.133497 22196 tablet_copy_client.cc:538] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:58:57.142303 22196 tablet_bootstrap.cc:492] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:58:57.161113 22196 tablet_bootstrap.cc:492] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:58:57.162040 22196 tablet_bootstrap.cc:492] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 02:58:57.162674 22196 ts_tablet_manager.cc:1397] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20250116 02:58:57.165096 22196 raft_consensus.cc:357] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:57.166044 22196 raft_consensus.cc:738] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 02:58:57.166733 22196 consensus_queue.cc:260] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:57.172616 22196 ts_tablet_manager.cc:1428] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250116 02:58:57.174929 20766 tablet_copy_service.cc:342] P 17446c8dacf04461acb843547665250f: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-bf7916e71bc94e1eba4ab8fd75257e48 received from {username='slave'} at 127.0.0.1:35540
I20250116 02:58:57.175410 20766 tablet_copy_service.cc:434] P 17446c8dacf04461acb843547665250f: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-bf7916e71bc94e1eba4ab8fd75257e48 on tablet bf7916e71bc94e1eba4ab8fd75257e48 with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:58:57.193614 21332 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:57.194929 22190 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:57.217871 20608 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:57.219354 20677 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:57.219409 22119 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:57.225225 22163 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:57.267899 21332 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:57.274210 22119 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:57.311596 21332 raft_consensus.cc:1212] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250116 02:58:57.390682 21332 raft_consensus.cc:1212] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250116 02:58:57.572918 21334 raft_consensus.cc:1212] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250116 02:58:57.618398 21334 raft_consensus.cc:1212] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250116 02:58:57.625541 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:57.681350 20683 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:58:57.682129 20683 raft_consensus.cc:604] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:58:57.687043 20608 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96dc04669716445ab04931358be7efbf"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
mode: GRACEFUL
new_leader_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:46758
I20250116 02:58:57.687848 20608 raft_consensus.cc:604] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Received request to transfer leadership to TS 231d91433a404468a44ea0a107345aeb
I20250116 02:58:57.691047 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250116 02:58:57.693264 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:58:57.693976 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:58:57.919298 22157 raft_consensus.cc:1059] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:57.922044 22157 consensus_queue.cc:237] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:57.936955 21334 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:57.937976 20677 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:57.938647 20749 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:57.939498 22190 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:57.940555 22163 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:57.947396 22213 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:58:57.963505 22157 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:57.965049 20683 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:57.980835 21332 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:57.985726 20503 catalog_manager.cc:5526] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb reported cstate change: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:57.987303 20753 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.007674 22191 raft_consensus.cc:1059] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:58.012565 22191 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:58.027590 21332 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:58.027545 20601 consensus_queue.cc:237] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:58.030092 20606 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:58.030416 20753 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:58.030978 22119 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:58.033106 22119 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:58.035699 22185 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:58:58.037760 20683 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:58.041118 21332 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:58.043709 22213 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:58.046902 22157 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:58:58.061554 22163 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.078310 20493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 2960f90503b44912b4b4b58cd2ceb89f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:58.080652 22191 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.082448 20677 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.086162 20608 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.091259 22185 raft_consensus.cc:988] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:58.091842 22185 raft_consensus.cc:1076] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 17446c8dacf04461acb843547665250f to start an election
I20250116 02:58:58.091701 20505 catalog_manager.cc:5526] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 4 to 5, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:58.094938 20756 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2"
dest_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:50656
I20250116 02:58:58.095620 20756 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:58.096094 20756 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:58.107194 20749 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.109076 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet 2960f90503b44912b4b4b58cd2ceb89f with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50634
I20250116 02:58:58.112031 22157 raft_consensus.cc:988] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: : Instructing follower 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:58.112681 22213 raft_consensus.cc:1076] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Signalling peer 231d91433a404468a44ea0a107345aeb to start an election
I20250116 02:58:58.115516 20683 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96dc04669716445ab04931358be7efbf"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
from {username='slave'} at 127.0.0.1:34866
I20250116 02:58:58.116124 20683 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:58:58.116498 20683 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:58.123992 21334 raft_consensus.cc:2949] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.137153 20683 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:58.141189 20756 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:58.168329 20756 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:58.143726 20505 catalog_manager.cc:5526] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.180161 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:58.180984 20608 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:58.149351 21332 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.150676 20683 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:58.151940 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:58:58.181747 20607 raft_consensus.cc:3049] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:58.153899 22227 tablet_replica.cc:331] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: stopping tablet replica
I20250116 02:58:58.182112 20607 raft_consensus.cc:738] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:58:58.183243 22227 raft_consensus.cc:2238] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:58:58.182904 20607 consensus_queue.cc:260] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:58.184123 22227 raft_consensus.cc:2267] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:58:58.184263 20607 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:58.154278 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:58:58.187314 20749 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:58.189263 20608 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:58.193893 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:58:58.194561 20683 raft_consensus.cc:3049] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:58:58.195070 20683 raft_consensus.cc:738] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:58.195166 20607 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:58.195175 20749 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:58.196388 20649 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:58.196552 20723 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:58:58.197999 22157 raft_consensus.cc:1059] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:58.199489 20683 consensus_queue.cc:260] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:58.203372 22227 ts_tablet_manager.cc:1905] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:58:58.202112 22157 consensus_queue.cc:237] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:58.204077 22191 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:58.209204 22191 raft_consensus.cc:695] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:58:58.210201 22191 consensus_queue.cc:237] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:58.213330 20683 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:58:58.216332 20677 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:58.218884 22228 raft_consensus.cc:2798] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:58:58.238039 21332 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:58.240953 20683 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:58:58.249416 22119 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:58.246876 22228 raft_consensus.cc:695] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:58:58.268115 21334 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:58.273470 20608 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:58:58.279489 22226 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:58:58.279165 22228 consensus_queue.cc:237] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:58.281261 22117 raft_consensus.cc:1059] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
I20250116 02:58:58.300427 21332 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.297245 22226 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.299921 22190 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20250116 02:58:58.312125 20683 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:58:58.315644 20608 raft_consensus.cc:2949] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:58:58.320528 22227 ts_tablet_manager.cc:1918] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250116 02:58:58.321067 22227 log.cc:1198] T 2960f90503b44912b4b4b58cd2ceb89f P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/2960f90503b44912b4b4b58cd2ceb89f
I20250116 02:58:58.322194 20490 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 38583cae882144f48cec12c72ba697cf with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250116 02:58:58.327786 22190 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250116 02:58:58.365695 22230 log.cc:926] Time spent T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20250116 02:58:58.373252 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 118707us
I20250116 02:58:58.830473 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250116 02:58:59.052130 20723 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.797s user 0.038s sys 0.022s
W20250116 02:58:59.052829 20723 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.798s user 0.038s sys 0.022s
W20250116 02:58:59.058225 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 803284us
I20250116 02:58:59.175293 22119 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.176103 22119 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:58:59.180344 22119 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.184036 22119 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.184726 22119 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.188045 22119 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.215308 22226 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.281522 22226 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.285127 22226 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.308912 22211 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.309944 22211 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.314019 22211 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.373823 22225 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.519904 22225 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.522239 22225 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.523038 22225 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.523646 22225 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.430012 22119 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.441447 22240 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.418072 22241 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.460135 22232 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.461102 22226 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.471868 22156 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.518518 22243 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.526098 22225 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.526393 22242 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.538297 22211 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.540890 22245 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.585749 22244 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.524649 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet 2960f90503b44912b4b4b58cd2ceb89f (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
W20250116 02:58:59.670220 20491 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 145598us
I20250116 02:58:59.625782 22248 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.654681 22252 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.655822 22247 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.658246 22225 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.621290 22119 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.636334 22240 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.678830 22247 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.686120 22250 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.686187 22254 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.676379 22244 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.686889 22250 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.686934 22254 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.687937 22240 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.688252 22244 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:59.689043 22244 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.689538 22244 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.689867 22250 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:59.689869 22254 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.635973 22232 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.647711 22226 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.669667 22211 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.656567 22156 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.679330 22252 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.680277 22248 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.701948 22247 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:59.701936 22244 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.656356 22242 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.692016 22256 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.702487 22248 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.668820 22243 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.762333 22244 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.763007 22248 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.697508 22117 consensus_queue.cc:237] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.764137 22248 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.705607 22119 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.766498 22119 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.707664 22211 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.747386 22259 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.691633 22225 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.706799 22226 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.709024 22232 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.705791 22242 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.773499 22232 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.705708 22156 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.676237 22245 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.764184 22244 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.632407 22241 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.778724 22253 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.709470 22246 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.702670 22252 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.710080 22256 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.711025 22258 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.796077 22255 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.711062 22257 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.774521 22232 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.762254 22240 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.777988 22226 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.779809 22253 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.768137 22259 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.815514 22244 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.816939 22253 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.817198 22259 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.794353 22243 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.802173 22225 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.818149 22226 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.797224 22255 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.804423 22241 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.821198 22226 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.796164 22119 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.823813 22232 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.818217 22240 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.804531 22248 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.804476 20504 catalog_manager.cc:5526] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:59.808178 22245 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.797232 22258 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.790329 22246 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.827364 22249 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.827874 22119 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.831041 22225 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.832213 22240 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.832018 22225 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.821250 22255 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.829284 22259 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.812614 22257 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.835719 22240 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.828253 22249 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.845811 22255 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.835973 22259 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.877859 22156 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.881399 22260 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.881974 22225 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.874246 22255 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.886195 22156 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.846791 22245 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.889283 22156 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.874233 22246 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:58:59.845201 22243 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.892769 22256 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.880365 22264 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.895756 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:58:59.900903 22258 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.902570 22249 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.834434 22232 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.847018 22248 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.896389 22264 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.907102 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:58:59.899152 22156 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:58:59.888089 22245 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.904495 22240 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.847229 22119 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.912689 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:58:59.936429 21332 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:59.937944 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:50634 (request call id 597) took 1653 ms. Trace:
I20250116 02:58:59.882419 22260 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.938346 22240 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.921658 22253 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.907991 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:58:59.923981 22267 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.926267 22241 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:58:59.926790 22264 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.930174 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:58:59.931638 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:58:59.932590 22156 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.942376 21336 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:59.940217 22253 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.943269 22232 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.947883 22241 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.948499 22225 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.957095 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:58:59.943238 22267 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.931721 22259 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.945897 22260 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.957950 22225 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.960925 21335 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:58:59.961987 22257 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.958981 22240 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:58:59.937978 22119 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.958904 22243 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.993779 22117 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:58:59.960847 22248 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:58:59.978446 22240 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:58:59.991889 20571 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:00.376786 22240 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:00.377512 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 402991us
I20250116 02:58:59.957177 20646 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:00.381104 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 425128us
I20250116 02:59:00.381341 22240 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:00.382126 22240 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.382820 22240 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:00.384155 20646 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:58:59.995116 22267 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.385798 22267 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:00.386180 22267 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.386511 22267 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:58:59.995878 22245 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.960177 22255 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:58:59.997785 22257 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.994922 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:58:59.993315 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.464468 21334 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:58:59.941103 20752 rpcz_store.cc:276] 0116 02:58:58.283921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:58:58.284113 (+ 192us) service_pool.cc:225] Handling call
0116 02:58:59.936601 (+1652488us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b5400370ad4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c94d50 00007fc326d17f2a 00007fc32bc71c7e 00007fc32534e817 00007fc32534e7a2 00007fc32534e6d1 00007fc32534e64c 00007fc32534d902 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:58:59.937624 (+ 1023us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3792081664}
I20250116 02:59:00.007721 22242 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.465793 22242 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:00.466812 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.467681 22292 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:00.468439 22292 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.008159 22243 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:00.010113 22253 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.991256 22156 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.996897 22225 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.011667 22258 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.487716 22258 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.490166 22258 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.013643 22249 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.501909 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.014212 22266 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.014456 22259 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:58:59.990648 22119 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:58:59.991428 22232 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.019819 22246 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.523078 22246 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } attrs { replace: true } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.017499 22264 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.038929 22252 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.251745 22273 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.262766 22276 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.267052 22272 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.271020 22256 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.271835 22270 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.272578 22274 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.525923 22264 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.272605 22275 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:00.527297 22246 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.275997 22278 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.283326 22281 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:00.287860 22280 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:00.307782 22279 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.308462 22282 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:00.329751 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:00.330513 20608 consensus_meta.cc:287] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 2.013s user 0.019s sys 0.031s
I20250116 02:59:00.335484 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.336833 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.339279 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.340596 22285 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.340958 22260 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.532025 22285 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.532251 22260 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.532778 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.339759 22262 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.347029 22283 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.349650 22277 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:00.350838 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 43858.3us
I20250116 02:59:00.367091 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:00.315426 20722 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.921s user 0.004s sys 0.024s
I20250116 02:59:00.376726 22248 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.389233 22257 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.605389 22255 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.374361 22117 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.606194 22255 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.608336 22117 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.608470 22255 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.414340 22267 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.609706 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.609999 22257 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.435613 22240 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.605325 22288 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:00.439812 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 96316.1us
I20250116 02:59:00.457005 22294 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:00.007896 22241 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.469290 22292 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:00.470045 20749 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:59:00.468747 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.616292 20752 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:00.470973 22242 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:00.474433 20504 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41972 (request call id 334) took 2201 ms. Trace:
I20250116 02:59:00.486145 22243 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.617620 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.012794 22263 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.491444 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.619109 21335 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:00.619688 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.492717 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.492769 20604 raft_consensus.cc:1270] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:59:00.494247 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.494488 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.495602 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:00.622411 20722 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.251s user 0.010s sys 0.025s
I20250116 02:59:00.622977 20600 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:00.623159 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.623514 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:00.497016 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.624378 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.625098 20604 raft_consensus.cc:2463] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:00.626168 22242 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:00.499229 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.500856 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.501336 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.490178 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.503389 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.503707 22249 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.507898 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.502979 22266 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.506336 22259 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.505627 22226 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.632002 22226 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.632881 22249 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.539121 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.540306 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.542492 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.526041 22270 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:00.527130 22274 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.535406 22285 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.544715 22293 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.533617 22281 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.529855 22273 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.530139 22280 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.544588 22283 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.544240 22277 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.644169 20751 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:00.574934 20505 catalog_manager.cc:5526] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 4 to 5, VOTER 17446c8dacf04461acb843547665250f (127.18.238.131) evicted. New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:00.575196 22264 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.533206 22279 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.545238 22276 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.552274 22278 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.550756 22252 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.549922 22256 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.572748 22232 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:00.546509 22282 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.528925 22275 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.594098 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.626875 22242 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.550963 22272 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.549647 22262 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.600060 22260 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.601652 21332 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEARNER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:59:00.603222 22245 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:00.604120 22253 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.611311 22288 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.613178 22257 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.610492 22267 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.611573 22294 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.615240 22240 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.626101 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.499046 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.620175 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:00.624262 21336 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:00.627861 20646 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:00.630250 21333 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:59:00.630996 20602 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:00.510840 22295 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.613459 22297 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:00.570138 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.645473 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.624322 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.626910 20754 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:00.648521 20756 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:00.622354 22290 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:00.649371 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1649) took 2330 ms. Trace:
I20250116 02:59:00.653059 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.633270 22258 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:00.618544 22263 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.678709 22156 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.656941 20606 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:00.658316 20753 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:00.618090 22243 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.658366 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.664242 22232 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.657976 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.662441 22264 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:00.678076 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 50302.5us
I20250116 02:59:00.685631 20756 raft_consensus.cc:1270] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:59:00.668256 22253 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.698688 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.684702 22260 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:00.705633 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.707026 21300 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:00.684407 20600 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:00.709915 21300 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:00.666090 22293 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:00.710752 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.687436 22156 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.693713 22266 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.617846 20504 rpcz_store.cc:276] 0116 02:58:58.272986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:58:58.273621 (+ 635us) service_pool.cc:225] Handling call
0116 02:59:00.474311 (+2200690us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:58:59.917354 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:58:59.932623 (+ 15269us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:58:59.932676 (+ 53us) write_op.cc:435] Acquired schema lock
0116 02:58:59.932716 (+ 40us) tablet.cc:659] Decoding operations
0116 02:58:59.933341 (+ 625us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:58:59.933439 (+ 98us) write_op.cc:641] Partition lock acquired for write op
0116 02:58:59.933470 (+ 31us) tablet.cc:682] Acquiring locks for 1 operations
0116 02:58:59.934047 (+ 577us) tablet.cc:698] Row locks acquired
0116 02:58:59.934084 (+ 37us) write_op.cc:260] PREPARE: finished
0116 02:58:59.934326 (+ 242us) write_op.cc:270] Start()
0116 02:58:59.934470 (+ 144us) write_op.cc:276] Timestamp: P: 1736996339934258 usec, L: 0
0116 02:58:59.934507 (+ 37us) op_driver.cc:348] REPLICATION: starting
0116 02:58:59.935458 (+ 951us) log.cc:844] Serialized 544 byte log entry
0116 02:59:00.320036 (+384578us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b54003811a0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326c9d9d1 00007fc326ca8480 00007fc326ca8428 00007fc326ca83a8 00007fc326ca8363 00007fc326ca75ec 00007fc32cadfbb6 00007fc32cada568 00007fc326ce421b 00007fc326ce6d10
0116 02:59:00.348248 (+ 28212us) op_driver.cc:464] REPLICATION: finished
0116 02:59:00.457458 (+109210us) write_op.cc:301] APPLY: starting
0116 02:59:00.457565 (+ 107us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:00.457682 (+ 117us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:00.457728 (+ 46us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:00.458068 (+ 340us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:00.458437 (+ 369us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0116 02:59:00.458505 (+ 68us) write_op.cc:312] APPLY: finished
0116 02:59:00.459133 (+ 628us) log.cc:844] Serialized 23 byte log entry
0116 02:59:00.460128 (+ 995us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:00.460235 (+ 107us) write_op.cc:454] Released schema lock
0116 02:59:00.460314 (+ 79us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:00.461314 (+ 1000us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:00.461351 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":60234,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":106784,"apply.run_cpu_time_us":2278,"apply.run_wall_time_us":5917,"num_ops":1,"prepare.queue_time_us":84406,"prepare.run_cpu_time_us":7697,"prepare.run_wall_time_us":109835,"raft.queue_time_us":26960,"raft.run_cpu_time_us":40184,"raft.run_wall_time_us":104386,"replication_time_us":413542,"spinlock_wait_cycles":9234304,"thread_start_us":241638,"threads_started":4,"wal-append.queue_time_us":88908}]]}
I20250116 02:59:00.709182 22257 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.724263 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.724681 22211 raft_consensus.cc:2798] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:00.712165 20603 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:00.692359 20599 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:00.705382 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.683530 20608 rpcz_store.cc:276] 0116 02:58:58.314711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:58:58.315010 (+ 299us) service_pool.cc:225] Handling call
0116 02:58:58.315338 (+ 328us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:58:58.315518 (+ 180us) raft_consensus.cc:1529] Early marking committed up to index 5
0116 02:59:00.537532 (+2222014us) log.cc:844] Serialized 14 byte log entry
0116 02:59:00.605075 (+ 67543us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:00.617067 (+ 11992us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:00.617115 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:00.620545 (+ 3430us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:00.645481 (+ 24936us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":939,"raft.queue_time_us":2356,"raft.run_cpu_time_us":396,"raft.run_wall_time_us":7346,"thread_start_us":28015,"threads_started":1,"wal-append.queue_time_us":83072}
I20250116 02:59:00.726676 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.704214 22226 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.708144 22260 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.724388 22126 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:00.736248 22283 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.736688 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.738005 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.738166 22256 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:00.732108 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:00.711434 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.711920 22225 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:00.736688 22156 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:00.682374 22290 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.706041 22245 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.753034 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.731530 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.682562 22297 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.743180 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.725510 22277 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:00.736289 22253 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.731971 22211 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.738437 22252 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.740598 22272 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.724247 22274 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.750890 22273 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.768193 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.768471 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.746889 22260 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:00.769215 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.784080 20605 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:00.750303 22126 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:00.709165 22295 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.758702 22225 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:00.775338 22275 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.777228 22245 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.777951 20649 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:00.779172 20677 raft_consensus.cc:1235] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:00.727190 22264 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.730082 22270 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.726276 22258 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:00.785602 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:00.756444 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.758772 20503 catalog_manager.cc:5526] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:59:00.975020 20607 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:00.976125 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.978080 20753 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:00.932713 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.959197 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.932250 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.783991 22211 raft_consensus.cc:3054] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:00.881489 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:00.976928 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:00.762413 22288 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.750520 22280 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.882187 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:00.989646 20604 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:00.880343 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:00.991503 20683 raft_consensus.cc:2463] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:00.991420 21336 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:00.764024 22282 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.881058 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:00.979907 20757 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
W20250116 02:59:00.985368 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 299163us
I20250116 02:59:00.730271 22300 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:00.772904 22259 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.775602 22242 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:00.784073 22263 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.875614 22281 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.976714 22225 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:00.994383 20646 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:00.783033 22243 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:00.990931 20681 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:59:00.932072 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.877148 22257 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.874858 22126 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.996652 20600 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:01.004868 22290 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.003782 22245 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:01.007157 20649 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:00.889350 22276 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:00.884409 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:00.986372 21329 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
W20250116 02:59:01.015676 22299 log.cc:926] Time spent T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.264s user 0.000s sys 0.001s
I20250116 02:59:00.995652 22270 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:01.030807 22270 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.031344 20646 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:01.032191 22267 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:01.032533 22267 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:01.015180 20778 debug-util.cc:398] Leaking SignalData structure 0x7b08000f5860 after lost signal to thread 19389
I20250116 02:59:01.033097 22267 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:01.034283 20778 debug-util.cc:398] Leaking SignalData structure 0x7b080003c600 after lost signal to thread 20540
I20250116 02:59:01.021086 22293 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.035362 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.036077 20753 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:01.015439 22253 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
W20250116 02:59:01.035939 20778 debug-util.cc:398] Leaking SignalData structure 0x7b0800353020 after lost signal to thread 20632
I20250116 02:59:01.037351 22253 raft_consensus.cc:695] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:01.004509 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:00.780227 22294 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.876263 22279 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:01.023043 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.022514 22119 raft_consensus.cc:2798] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:01.040452 22119 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:01.040962 22119 raft_consensus.cc:3054] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:01.020655 22245 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:01.028610 20754 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
W20250116 02:59:01.020231 20647 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.754s (SENT) [suppressed 2 similar messages]
I20250116 02:59:01.039223 20749 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:01.027333 20649 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:01.047211 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.049176 20755 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:01.050292 20649 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:01.051210 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.052379 22298 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:01.052891 22298 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:01.053264 22298 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:01.036937 22270 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:00.873373 22296 consensus_queue.cc:1035] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.348s
I20250116 02:59:01.058635 22226 raft_consensus.cc:2798] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:01.050261 22245 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:01.061300 22226 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:01.061862 22226 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:01.084914 20649 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:01.086717 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 321847us
I20250116 02:59:01.041765 22297 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:00.931172 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.093719 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.094220 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.095220 20676 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
W20250116 02:59:01.087311 20778 debug-util.cc:398] Leaking SignalData structure 0x7b08003029e0 after lost signal to thread 20707
I20250116 02:59:01.093650 20649 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:01.100497 21300 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:01.091058 22248 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:01.105858 22295 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.102921 22248 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:01.110528 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.084173 22278 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:01.024518 22300 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:01.102474 20778 debug-util.cc:398] Leaking SignalData structure 0x7b08003031e0 after lost signal to thread 20708
I20250116 02:59:01.110059 22248 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:01.115746 22243 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250116 02:59:01.122527 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 121987us
I20250116 02:59:01.087538 22211 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.114874 22262 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.124812 22258 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.109431 22288 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:01.125864 22243 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:01.130402 21300 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:01.130699 22300 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:01.132393 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.133762 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:01.138226 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 88225.9us
I20250116 02:59:01.139083 20649 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:01.147141 22303 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:01.148226 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.151207 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:01.148936 22232 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:01.135188 22243 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:01.152556 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.134704 22264 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:01.149545 20778 debug-util.cc:398] Leaking SignalData structure 0x7b08002f00a0 after lost signal to thread 20781
I20250116 02:59:01.162916 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:01.160051 22292 consensus_meta.cc:287] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.541s user 0.005s sys 0.009s
I20250116 02:59:01.149149 22260 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.118811 22253 consensus_queue.cc:237] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.157164 22240 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:01.166947 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:01.164637 20778 debug-util.cc:398] Leaking SignalData structure 0x7b0800354e00 after lost signal to thread 20782
I20250116 02:59:01.169832 22240 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:01.165385 22289 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:01.175761 20682 raft_consensus.cc:2949] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:01.176895 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.179428 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.179477 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.174098 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.180750 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.178952 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.181011 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.159112 22303 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:01.181246 20677 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:01.185242 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.183418 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.180701 22289 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.187943 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.188608 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.190045 20675 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:01.190125 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.190371 20681 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:01.177690 22240 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:01.164034 22267 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.196957 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.198853 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.199779 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.186085 22211 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.189008 22303 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:01.203637 22119 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.147558 22285 raft_consensus.cc:2949] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:01.205749 22119 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:01.210852 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.211427 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.213647 20681 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:59:01.167120 22292 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.215863 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.217974 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:01.210217 20778 debug-util.cc:398] Leaking SignalData structure 0x7b080004ab60 after lost signal to thread 21360
W20250116 02:59:01.219673 20778 debug-util.cc:398] Leaking SignalData structure 0x7b08002218e0 after lost signal to thread 21361
I20250116 02:59:01.219141 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.222493 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.223965 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.224980 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.225143 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.226101 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.227537 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.227798 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.222702 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.222865 22225 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.223325 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.224164 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.226303 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.183400 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.236547 20604 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:01.237038 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.236409 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.236493 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.239122 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.239619 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.240811 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.241432 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.241922 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.247479 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.248948 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.249068 20720 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:01.261922 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.265712 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.251052 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:01.248098 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.247462 22276 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20250116 02:59:01.253012 22126 raft_consensus.cc:2798] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:01.275169 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.279321 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:01.280150 21327 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:01.275214 21328 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 2: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:01.281697 20676 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:01.283795 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.270910 22260 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:01.308858 20608 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250116 02:59:01.310451 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.281158 22126 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:01.321549 20682 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:01.281415 20599 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:01.324357 22126 raft_consensus.cc:3054] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:01.348722 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:01.358912 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:01.363564 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 206966us
I20250116 02:59:01.337288 22267 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:01.364657 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.366597 20682 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:01.343011 22289 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.328428 22240 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.373246 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.374306 20649 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:01.374815 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:01.375362 20649 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:01.375752 22255 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:01.376184 22255 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:01.376567 22255 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:01.376842 22119 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:01.377223 22119 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:01.377549 22119 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:01.362481 22292 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:01.376238 20649 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:01.362799 22225 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.382990 20649 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:01.383404 22306 raft_consensus.cc:2798] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:01.383098 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:01.383750 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.390998 21334 raft_consensus.cc:1235] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:01.391112 20649 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:01.396466 20677 raft_consensus.cc:2388] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:01.397526 20608 raft_consensus.cc:1270] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:59:01.400547 22240 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:01.402397 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.404420 21336 raft_consensus.cc:1235] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:01.407568 21328 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:01.405122 20647 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:01.390842 22306 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:01.395267 22267 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:01.415750 22119 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.417444 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.419570 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:01.421022 22248 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.415833 22267 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:01.423303 22267 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:01.423702 22288 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:01.424130 22288 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:01.424480 22288 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:01.425347 20676 raft_consensus.cc:1235] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:01.442795 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:01.428040 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.443645 20607 raft_consensus.cc:3054] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:01.444955 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:01.446753 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:01.447361 20605 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:01.447903 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.449179 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:01.454203 20607 raft_consensus.cc:2463] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:01.455238 20605 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:01.456876 22119 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.414724 22306 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:01.459506 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.464306 20676 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:01.459497 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.464476 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.414036 22313 consensus_queue.cc:1035] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:01.415894 20675 raft_consensus.cc:1235] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:01.459621 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.466428 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.467511 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.413506 22298 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.472862 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.474678 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.470132 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:01.475960 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 72755.3us
I20250116 02:59:01.476752 20647 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:01.477057 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.477809 20601 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:01.470656 22309 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:01.479094 22309 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.475652 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.463089 22241 raft_consensus.cc:2798] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:01.485730 22241 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:01.486164 22241 raft_consensus.cc:3054] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:01.490939 20649 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:01.492406 20680 raft_consensus.cc:2388] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 2: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:01.492623 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.490551 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.494822 20683 raft_consensus.cc:2388] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 2: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:01.495101 20649 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:01.416744 22243 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.426069 22245 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.499495 22119 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:01.512745 22119 raft_consensus.cc:695] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:01.504079 20674 raft_consensus.cc:2388] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:01.500751 22307 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:01.501056 22240 raft_consensus.cc:2798] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:01.525596 20679 raft_consensus.cc:2388] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:01.526110 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.500345 22248 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.524084 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.531797 22119 consensus_queue.cc:237] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.470394 22226 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.555011 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:01.864943 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 369934us
I20250116 02:59:01.664469 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.808218 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.560596 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.776808 20503 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41978 (request call id 323) took 1088 ms. Trace:
I20250116 02:59:01.809044 20505 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48594 (request call id 216) took 2284 ms. Trace:
W20250116 02:59:01.869695 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 290372us
W20250116 02:59:01.870512 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 344508us
I20250116 02:59:01.610316 21327 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:01.872072 21330 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:01.872597 20646 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:01.872900 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.873641 22260 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:01.867039 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.685674 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.874194 22260 raft_consensus.cc:695] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:01.874457 20605 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:01.864645 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.875439 22260 consensus_queue.cc:237] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.876266 20601 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:01.873152 20647 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:01.883834 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.893966 20683 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:01.893795 20647 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:01.533510 22276 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:01.854202 22243 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.850466 22264 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:01.896449 20678 raft_consensus.cc:2388] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 2: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:01.605336 22240 raft_consensus.cc:695] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:01.899642 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:01.779502 22298 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.634411 22309 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:01.901530 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:01.817219 22245 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.818977 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:43834 (request call id 804) took 1297 ms. Trace:
I20250116 02:59:01.858095 22241 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.665923 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.902506 22253 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:01.710491 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:01.904729 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 308619us
I20250116 02:59:01.827702 22255 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:01.611009 22126 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.884166 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:01.905737 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1440) took 1023 ms. Trace:
I20250116 02:59:01.905943 22240 consensus_queue.cc:237] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.868618 20503 rpcz_store.cc:276] 0116 02:59:00.624991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:00.625242 (+ 251us) service_pool.cc:225] Handling call
0116 02:59:01.712678 (+1087436us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:01.202352 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:01.202902 (+ 550us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:01.202931 (+ 29us) write_op.cc:435] Acquired schema lock
0116 02:59:01.202947 (+ 16us) tablet.cc:659] Decoding operations
0116 02:59:01.203280 (+ 333us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:01.203327 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:01.203351 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0116 02:59:01.203536 (+ 185us) tablet.cc:698] Row locks acquired
0116 02:59:01.203550 (+ 14us) write_op.cc:260] PREPARE: finished
0116 02:59:01.203724 (+ 174us) write_op.cc:270] Start()
0116 02:59:01.203846 (+ 122us) write_op.cc:276] Timestamp: P: 1736996341203656 usec, L: 0
0116 02:59:01.203862 (+ 16us) op_driver.cc:348] REPLICATION: starting
0116 02:59:01.204496 (+ 634us) log.cc:844] Serialized 544 byte log entry
0116 02:59:01.442512 (+238016us) op_driver.cc:464] REPLICATION: finished
0116 02:59:01.694881 (+252369us) write_op.cc:301] APPLY: starting
0116 02:59:01.694963 (+ 82us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:01.695091 (+ 128us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:01.695203 (+ 112us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:01.695520 (+ 317us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:01.706739 (+ 11219us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0116 02:59:01.706806 (+ 67us) write_op.cc:312] APPLY: finished
0116 02:59:01.707098 (+ 292us) log.cc:844] Serialized 23 byte log entry
0116 02:59:01.711159 (+ 4061us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:01.711260 (+ 101us) write_op.cc:454] Released schema lock
0116 02:59:01.711306 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:01.711684 (+ 378us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:01.711713 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":252115,"apply.run_cpu_time_us":2029,"apply.run_wall_time_us":16762,"num_ops":1,"prepare.queue_time_us":198054,"prepare.run_cpu_time_us":2415,"prepare.run_wall_time_us":3354,"replication_time_us":238488,"thread_start_us":47129,"threads_started":1}]]}
I20250116 02:59:01.618093 22303 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:01.947988 20679 rpcz_store.cc:276] 0116 02:59:00.881988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:00.883920 (+ 1932us) service_pool.cc:225] Handling call
0116 02:59:01.905515 (+1021595us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54000875d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:01.905654 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1117380480}
I20250116 02:59:01.949880 22303 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.949818 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.901340 22276 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.908063 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.664355 22281 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:01.951181 22281 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.951809 20680 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:01.952564 22276 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:01.953052 20680 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:01.953173 22281 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:01.850172 22288 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.901005 22262 raft_consensus.cc:2798] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:01.954795 22262 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:01.955195 22262 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:01.956352 22126 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:01.915447 20601 raft_consensus.cc:3049] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:01.903223 20647 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:01.639413 22267 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:01.943614 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.165932 20681 raft_consensus.cc:2388] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:02.166105 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:02.166344 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:02.167045 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.168089 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:01.954138 20680 raft_consensus.cc:1235] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:02.168778 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:02.169430 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:02.169827 22320 log.cc:926] Time spent T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.230s user 0.001s sys 0.010s
I20250116 02:59:02.170173 22326 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:02.059124 22323 log.cc:926] Time spent T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.117s user 0.000s sys 0.003s
I20250116 02:59:02.171931 22241 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:02.132122 22262 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:02.132618 20599 raft_consensus.cc:1235] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:01.912951 20602 rpcz_store.cc:276] 0116 02:59:00.521610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:00.856971 (+335361us) service_pool.cc:225] Handling call
0116 02:59:01.007662 (+150691us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe7a6 00007fc32bcbe980 00007fc32bc71b60 00007fc32534e817 00007fc32534e7a2 00007fc32534e6d1
0116 02:59:01.818050 (+810388us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b54002d8a54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c94d50 00007fc326d17f2a 00007fc32bc71c7e 00007fc32534e817 00007fc32534e7a2 00007fc32534e6d1 00007fc32534e64c 00007fc32534d902 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:01.818860 (+ 810us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1830043648}
W20250116 02:59:02.175426 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 272248us
I20250116 02:59:01.958799 20601 raft_consensus.cc:738] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:02.133160 22306 consensus_meta.cc:287] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.648s user 0.013s sys 0.003s
I20250116 02:59:01.961802 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:01.909313 22264 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:02.141911 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.142725 20677 raft_consensus.cc:1235] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:01.980340 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.149503 22226 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:01.869189 20505 rpcz_store.cc:276] 0116 02:58:59.524403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:58:59.777624 (+253221us) service_pool.cc:225] Handling call
0116 02:59:01.808835 (+2031211us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:00.783920 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:01.104290 (+320370us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:01.106876 (+ 2586us) write_op.cc:435] Acquired schema lock
0116 02:59:01.107890 (+ 1014us) tablet.cc:659] Decoding operations
0116 02:59:01.134221 (+ 26331us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:01.134325 (+ 104us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:01.134356 (+ 31us) tablet.cc:682] Acquiring locks for 1 operations
0116 02:59:01.138183 (+ 3827us) tablet.cc:698] Row locks acquired
0116 02:59:01.138203 (+ 20us) write_op.cc:260] PREPARE: finished
0116 02:59:01.138372 (+ 169us) write_op.cc:270] Start()
0116 02:59:01.138499 (+ 127us) write_op.cc:276] Timestamp: P: 1736996341138326 usec, L: 0
0116 02:59:01.138519 (+ 20us) op_driver.cc:348] REPLICATION: starting
0116 02:59:01.184141 (+ 45622us) log.cc:844] Serialized 539 byte log entry
0116 02:59:01.334475 (+150334us) op_driver.cc:464] REPLICATION: finished
0116 02:59:01.621915 (+287440us) write_op.cc:301] APPLY: starting
0116 02:59:01.622105 (+ 190us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:01.622239 (+ 134us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:01.660634 (+ 38395us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:01.664307 (+ 3673us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:01.664481 (+ 174us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0116 02:59:01.664527 (+ 46us) write_op.cc:312] APPLY: finished
0116 02:59:01.685476 (+ 20949us) log.cc:844] Serialized 23 byte log entry
0116 02:59:01.686414 (+ 938us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:01.686509 (+ 95us) write_op.cc:454] Released schema lock
0116 02:59:01.686553 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:01.694446 (+ 7893us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:01.694477 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":193994,"apply.run_cpu_time_us":2869,"apply.run_wall_time_us":86247,"num_ops":1,"prepare.queue_time_us":71438,"prepare.run_cpu_time_us":5591,"prepare.run_wall_time_us":437361,"replication_time_us":171677,"thread_start_us":82184,"threads_started":2}]]}
I20250116 02:59:02.151113 20740 tablet_service.cc:1514] Processing DeleteTablet for tablet 38583cae882144f48cec12c72ba697cf with delete_type TABLET_DATA_TOMBSTONED (TS 17446c8dacf04461acb843547665250f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50634
I20250116 02:59:01.961941 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.003719 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.162281 20503 catalog_manager.cc:5526] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250116 02:59:02.072865 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 206319us
I20250116 02:59:02.168186 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.944674 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.170410 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:02.170825 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.130344 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:01.958112 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.169454 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:02.051867 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 184883us
I20250116 02:59:02.191159 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:02.216749 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:02.217017 20755 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:02.193611 20601 consensus_queue.cc:260] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:02.195323 22264 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:02.195711 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:02.195888 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:02.196380 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:02.179761 22306 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:02.192843 22326 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:02.199261 22243 consensus_queue.cc:1046] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250116 02:59:02.200227 20604 raft_consensus.cc:1235] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:02.200920 22262 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:02.202107 20647 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:02.241344 20647 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:02.202656 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.242339 20607 raft_consensus.cc:2388] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:02.202841 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:02.203490 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:02.204221 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:02.244454 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.245558 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.205098 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.205204 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.206048 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.212935 20678 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:02.218950 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.225142 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.179106 22248 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:02.225178 20601 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:02.205461 21327 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:02.231284 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:02.251765 21334 raft_consensus.cc:2388] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 2: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:02.233461 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.236356 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:02.239724 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 99019.7us
I20250116 02:59:02.248658 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.248940 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.248924 20720 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:02.272903 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.273152 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.274992 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:02.276598 20680 raft_consensus.cc:1235] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:02.277984 20680 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:02.281416 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.233807 21330 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:02.283303 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.287230 20677 raft_consensus.cc:1235] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:02.287251 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.287663 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:02.288146 20604 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:02.246832 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:02.289116 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.250370 22248 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:02.292779 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:02.292526 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.293193 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.307394 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.307530 20678 raft_consensus.cc:1235] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:02.298401 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.310171 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.310599 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.287708 22253 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:02.308316 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.320945 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:02.321091 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:02.321352 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.295608 22248 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:02.324556 20608 raft_consensus.cc:2388] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:59:02.325214 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 117830us
W20250116 02:59:02.326756 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 85527.7us
I20250116 02:59:02.327399 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:02.327504 20647 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:02.287058 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:02.285221 22264 consensus_queue.cc:1046] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20250116 02:59:02.252880 22309 consensus_queue.cc:1046] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20250116 02:59:02.331512 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.296015 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.319284 20680 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:02.335048 20680 raft_consensus.cc:604] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:59:02.336081 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.337312 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.338430 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:02.339551 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.328691 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:02.340369 21330 raft_consensus.cc:1235] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:02.340754 20754 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:02.340512 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.349885 20755 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
W20250116 02:59:02.332628 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 91765.9us
I20250116 02:59:02.350023 20647 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:02.355978 20601 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:02.326074 22329 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:02.355340 21299 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:02.360003 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.361011 20678 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
W20250116 02:59:02.357649 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 69001.6us
I20250116 02:59:02.367494 20647 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:02.354478 20674 raft_consensus.cc:1235] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:02.352182 22276 raft_consensus.cc:2743] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:02.353936 22253 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:02.360576 22329 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:02.320159 22296 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20250116 02:59:02.331241 22292 consensus_queue.cc:1046] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20250116 02:59:02.375653 22336 consensus_queue.cc:1046] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:02.369330 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.326248 22324 consensus_queue.cc:1046] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.138s
W20250116 02:59:02.389896 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 90557.5us
I20250116 02:59:02.370769 22260 raft_consensus.cc:2798] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:02.410455 20647 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:02.383006 22253 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:02.390690 22329 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:02.371863 22285 consensus_queue.cc:1046] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20250116 02:59:02.401538 22306 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:02.409530 22260 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:02.485347 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 59804.3us
I20250116 02:59:02.417670 22253 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:02.415024 22327 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250116 02:59:02.492612 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 81652.3us
W20250116 02:59:02.495142 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 139857us
W20250116 02:59:02.496188 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 90532.4us
I20250116 02:59:02.419631 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:02.497604 21299 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:02.508807 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:02.509709 20601 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:02.488490 22260 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:02.487524 22333 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:02.429316 22241 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20250116 02:59:02.517191 20720 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:02.489181 22347 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:02.490118 22292 raft_consensus.cc:3049] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
W20250116 02:59:02.519362 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 159317us
I20250116 02:59:02.496827 22327 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:02.512014 22326 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
W20250116 02:59:02.534468 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 163369us
I20250116 02:59:02.485999 22338 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.112s
I20250116 02:59:02.532902 22292 raft_consensus.cc:738] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:02.535864 22327 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:02.574050 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:02.606323 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 74736.2us
I20250116 02:59:02.543774 22264 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:02.540123 22303 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:02.623063 22354 consensus_queue.cc:1046] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:02.624413 22354 raft_consensus.cc:3049] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:02.624629 21300 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:02.611130 22340 tablet_replica.cc:331] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f: stopping tablet replica
W20250116 02:59:02.634514 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 224106us
I20250116 02:59:02.637066 20647 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:02.637378 22341 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:02.639277 22341 raft_consensus.cc:695] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:02.640120 20679 raft_consensus.cc:1235] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:02.640307 22341 consensus_queue.cc:237] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:02.642597 20647 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:02.642722 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 119020us
I20250116 02:59:02.539165 22335 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20250116 02:59:02.647040 20647 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:02.624799 22354 raft_consensus.cc:738] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:02.571468 22326 consensus_queue.cc:1046] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20250116 02:59:02.649264 20601 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:02.650808 22264 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:02.652325 20603 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:02.645783 22336 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:02.654996 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 109198us
I20250116 02:59:02.655887 20647 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:02.659354 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:02.666118 20647 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:02.669945 20647 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:02.672673 20649 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:02.645207 22306 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:02.626960 22352 consensus_queue.cc:1046] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20250116 02:59:02.680584 22350 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:02.667749 22296 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:02.681360 20647 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:02.685930 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 153983us
I20250116 02:59:02.676962 22241 raft_consensus.cc:2743] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:02.704975 22255 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:02.709098 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 198071us
I20250116 02:59:02.683706 22333 raft_consensus.cc:2798] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:02.738756 20676 raft_consensus.cc:1235] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:02.678937 22356 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:02.698271 22306 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:02.680858 22358 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:02.677284 22264 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:02.749212 22356 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:02.754503 22349 log.cc:926] Time spent T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20250116 02:59:02.754993 22356 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:02.666692 22292 consensus_queue.cc:260] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:02.704975 22298 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20250116 02:59:02.757639 22253 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:02.801220 20720 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:02.755427 22354 consensus_queue.cc:260] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:02.743098 22355 consensus_queue.cc:1046] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
I20250116 02:59:02.833684 21334 raft_consensus.cc:1235] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:02.812319 22253 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:02.807585 22354 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:02.806478 22288 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:02.813340 22355 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:02.816243 22276 raft_consensus.cc:3049] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
W20250116 02:59:02.847671 22248 consensus_meta.cc:287] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.515s user 0.012s sys 0.000s
W20250116 02:59:02.862670 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 179007us
I20250116 02:59:02.853132 22276 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
W20250116 02:59:02.889730 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 202122us
I20250116 02:59:02.891304 22355 consensus_queue.cc:1035] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:02.896988 20723 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:02.897315 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 399718us
I20250116 02:59:02.898728 22324 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:02.899545 21299 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:02.901890 20723 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:02.901890 21299 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:02.904909 20723 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:02.877455 20720 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:02.853783 22253 consensus_queue.cc:1046] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
W20250116 02:59:02.949488 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 47685.5us
I20250116 02:59:02.950502 21299 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:02.868760 22248 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:02.965710 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 243467us
I20250116 02:59:02.975262 20682 raft_consensus.cc:2388] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:02.953706 22352 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:02.954170 22359 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:02.947978 22330 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:02.956070 22264 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:02.947942 22363 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:02.983227 22364 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:02.949208 22362 consensus_queue.cc:1046] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.107s
I20250116 02:59:02.995100 21329 raft_consensus.cc:1235] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:02.991618 22359 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.002003 22347 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:02.993554 22264 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:02.981945 22276 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.010756 22303 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.015039 22264 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:03.020979 22364 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:02.998986 22341 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:02.999142 22376 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.036401 22276 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.037602 22327 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:02.991189 22330 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:03.034947 22260 consensus_meta.cc:287] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.511s user 0.011s sys 0.001s
I20250116 02:59:03.100945 22264 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:03.052611 22341 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:03.107234 22341 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.107741 22241 raft_consensus.cc:3049] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:03.108247 22241 raft_consensus.cc:738] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:03.106338 22327 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:03.110447 22241 consensus_queue.cc:260] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.112404 22288 consensus_queue.cc:1046] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:03.095454 22334 raft_consensus.cc:3049] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:03.165165 22324 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.167194 22330 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:03.098701 22267 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:03.167822 22329 consensus_meta.cc:287] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.681s user 0.004s sys 0.009s
I20250116 02:59:03.064038 22333 raft_consensus.cc:695] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:03.173595 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 296240us
I20250116 02:59:03.055810 22248 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:03.047438 22362 raft_consensus.cc:3049] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:03.181047 20720 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.047466 22359 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:03.092594 22260 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.174638 22334 raft_consensus.cc:738] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:03.191072 20600 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.013s sys 0.022s
I20250116 02:59:03.181028 22288 consensus_queue.cc:1225] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer 57fcfcd5ab01438c9a4baf8a61871aa6 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250116 02:59:03.192988 22362 raft_consensus.cc:738] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:03.195020 20602 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250116 02:59:03.200457 20600 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.548s user 0.013s sys 0.022s
W20250116 02:59:03.197124 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 497064us
W20250116 02:59:03.224509 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 274104us
I20250116 02:59:03.192034 22324 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.211722 22334 consensus_queue.cc:260] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.196653 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:03.227399 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 288439us
I20250116 02:59:03.234746 20723 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:03.235809 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 265172us
I20250116 02:59:03.190042 22329 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:03.238978 20572 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.243352 20572 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.248759 20572 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:03.263079 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 417055us
W20250116 02:59:03.270196 21302 leader_election.cc:336] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.534s (SENT)
I20250116 02:59:03.268551 20572 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:03.267192 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 386502us
I20250116 02:59:03.272559 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.273156 20603 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.273427 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.274012 20605 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:03.274701 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.274977 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.276232 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.276747 20604 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.256922 22334 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.241611 22264 consensus_queue.cc:1046] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20250116 02:59:03.223402 22356 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.241850 21328 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:03.277892 21302 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.312448 22362 consensus_queue.cc:260] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:03.333846 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:03.266857 22379 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.325866 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:03.302594 22380 consensus_queue.cc:1046] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250116 02:59:03.259025 22376 consensus_queue.cc:1046] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20250116 02:59:03.347322 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.350257 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:03.347718 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:03.347389 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 77215.8us
I20250116 02:59:03.348745 20646 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:03.352324 21302 leader_election.cc:336] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.692s (SENT)
I20250116 02:59:03.352347 22119 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:03.352869 21302 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.352975 22119 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:03.353684 22119 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:03.365471 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 44470.6us
W20250116 02:59:03.369206 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 134623us
I20250116 02:59:03.347519 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:03.344761 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:03.253959 22333 consensus_queue.cc:237] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:03.370574 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 75647.8us
I20250116 02:59:03.371289 20601 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:03.429064 20723 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.425485 20605 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:03.371685 20572 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.430480 20723 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.320544 22366 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:03.348318 22354 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250116 02:59:03.446197 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 121053us
W20250116 02:59:03.433640 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 62075.5us
I20250116 02:59:03.395707 22352 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.366843 20599 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:03.435740 22126 raft_consensus.cc:3049] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:03.437258 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.412670 22362 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:03.390717 20674 raft_consensus.cc:3049] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:03.456156 20571 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:03.395429 22359 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.356371 22384 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.459626 20572 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.454702 22119 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.450639 22376 raft_consensus.cc:3049] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:03.466356 20649 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:03.476544 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.467056 21327 raft_consensus.cc:1235] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:03.441999 20748 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:03.482002 20571 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:03.482932 22119 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:03.437425 22385 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.461221 22329 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:03.461262 22126 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:03.464497 20674 raft_consensus.cc:738] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:03.469651 22390 raft_consensus.cc:2743] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.437884 22366 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:03.490197 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.493151 22387 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.478466 22384 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:03.470990 22376 raft_consensus.cc:738] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:03.480310 22260 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:03.498279 21300 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:03.467410 22253 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
W20250116 02:59:03.505446 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 197446us
I20250116 02:59:03.513871 20603 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:03.497470 22366 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.501438 22384 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:03.521522 20649 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:03.506186 22356 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:03.509934 22309 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:03.506148 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:03.529171 22373 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:03.519737 22385 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:03.536494 21328 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.539744 22374 raft_consensus.cc:2743] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.541249 22357 consensus_queue.cc:1046] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
W20250116 02:59:03.546219 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 193908us
W20250116 02:59:03.550776 21302 leader_election.cc:336] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.536s (SENT)
I20250116 02:59:03.551270 21302 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.530113 22260 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:03.530786 22309 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:03.526522 22384 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:59:03.594966 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 391573us
I20250116 02:59:03.530321 22387 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:03.546803 22385 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:03.598651 20720 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.601372 20604 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:03.606175 22389 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.596695 22309 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.604734 20649 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:03.592262 22373 raft_consensus.cc:695] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:03.607074 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 56353.7us
I20250116 02:59:03.598421 22387 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:03.612869 21302 leader_election.cc:336] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.629s (SENT)
I20250116 02:59:03.613358 21302 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:03.611613 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 128283us
I20250116 02:59:03.599073 22260 raft_consensus.cc:695] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:03.614689 20572 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:03.616019 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 134098us
I20250116 02:59:03.616138 22369 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:03.616760 22369 raft_consensus.cc:695] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:03.617715 22369 consensus_queue.cc:237] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:03.603087 22385 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:03.622929 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.624990 20572 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:03.630681 22386 log.cc:926] Time spent T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20250116 02:59:03.630895 20571 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:03.631016 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1716) took 1122 ms. Trace:
I20250116 02:59:03.636312 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.551100 22126 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.621913 22374 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.637455 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.615181 22387 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.602483 22341 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.643074 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.646296 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.592767 22376 consensus_queue.cc:260] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:03.651589 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.547878 20674 consensus_queue.cc:260] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.646467 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.653185 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.654281 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.654332 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.655256 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:03.649933 20571 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:03.655845 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.658679 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.631568 20600 rpcz_store.cc:276] 0116 02:59:02.508938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.511273 (+ 2335us) service_pool.cc:225] Handling call
0116 02:59:02.539720 (+ 28447us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:02.572420 (+ 32700us) raft_consensus.cc:1529] Early marking committed up to index 2
0116 02:59:03.234651 (+662231us) log.cc:844] Serialized 14 byte log entry
0116 02:59:03.625741 (+391090us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:03.628387 (+ 2646us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:03.629691 (+ 1304us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:03.630754 (+ 1063us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:03.630909 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":987,"prepare.queue_time_us":635976,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":2715,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":134,"thread_start_us":946953,"threads_started":2}
I20250116 02:59:03.660804 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:03.661561 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 231156us
I20250116 02:59:03.656358 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.662573 20723 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.650125 21302 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.633090 21330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 594) took 1136 ms. Trace:
I20250116 02:59:03.661365 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.657241 20571 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:03.665459 21302 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.635974 22390 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.684666 22394 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.685700 22394 consensus_queue.cc:1046] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:03.647540 22379 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:03.651151 22374 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.696161 22126 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.660900 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:03.700721 22398 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.701239 22393 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:03.702775 22352 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.705979 20681 raft_consensus.cc:1235] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:03.668658 22347 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.704300 22390 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:03.717058 22390 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:03.717541 22390 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.716213 22401 consensus_queue.cc:1046] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20250116 02:59:03.661083 22395 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.720175 22292 raft_consensus.cc:3054] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:03.706382 22394 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.711202 20600 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:03.669799 21330 rpcz_store.cc:276] 0116 02:59:02.496770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.501686 (+ 4916us) service_pool.cc:225] Handling call
0116 02:59:03.023683 (+521997us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:03.023992 (+ 309us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:03.024525 (+ 533us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:03.024559 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:03.025196 (+ 637us) log.cc:844] Serialized 26 byte log entry
0116 02:59:03.471730 (+446534us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:03.478463 (+ 6733us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:03.478546 (+ 83us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:03.632721 (+154175us) raft_consensus.cc:1680] finished
0116 02:59:03.632743 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:03.633002 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":961,"raft.queue_time_us":13219,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":11004,"spinlock_wait_cycles":2969984,"thread_start_us":437939,"threads_started":1,"wal-append.queue_time_us":500762}
I20250116 02:59:03.714502 22393 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:03.738826 22333 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:03.730729 22394 raft_consensus.cc:3049] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:03.687263 22260 consensus_queue.cc:237] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:03.775774 22400 raft_consensus.cc:3049] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:03.776418 22400 raft_consensus.cc:738] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:03.672830 22373 consensus_queue.cc:237] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:03.786233 21335 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:03.750097 22393 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:03.783404 22276 consensus_meta.cc:287] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.729s user 0.011s sys 0.003s
I20250116 02:59:03.764843 22394 raft_consensus.cc:738] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
W20250116 02:59:03.789487 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 172061us
W20250116 02:59:03.784648 22396 log.cc:926] Time spent T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
W20250116 02:59:03.805069 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 142555us
I20250116 02:59:03.765318 22333 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:03.811502 20752 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:03.795899 22341 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:03.813959 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:03.820070 20723 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:03.827445 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 162063us
I20250116 02:59:03.828661 21302 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.807614 22276 consensus_queue.cc:1046] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20250116 02:59:03.786125 22400 consensus_queue.cc:260] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:03.867784 22276 raft_consensus.cc:3049] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:03.868228 22276 raft_consensus.cc:738] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:03.869117 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1720) took 1197 ms. Trace:
I20250116 02:59:03.871496 20608 rpcz_store.cc:276] 0116 02:59:02.671932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.672216 (+ 284us) service_pool.cc:225] Handling call
0116 02:59:02.688710 (+ 16494us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:03.171165 (+482455us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:03.171200 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:03.177657 (+ 6457us) log.cc:844] Serialized 26 byte log entry
0116 02:59:03.441479 (+263822us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:03.441517 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:03.441597 (+ 80us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:03.868904 (+427307us) raft_consensus.cc:1680] finished
0116 02:59:03.868926 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:03.869053 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10000,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":19804,"thread_start_us":727045,"threads_started":2,"wal-append.queue_time_us":325898}
I20250116 02:59:03.840039 22126 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:03.878238 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 49676.2us
I20250116 02:59:03.880079 21302 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.881644 20723 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.884548 20723 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:03.879900 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 222796us
I20250116 02:59:03.887789 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.893797 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.801507 22385 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:03.920473 20723 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.896054 21302 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.922183 20571 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:03.919106 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:03.924580 20678 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:03.928284 21302 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:03.867766 22395 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.931512 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:03.934974 20678 raft_consensus.cc:1235] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:03.986209 21302 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:03.877544 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:03.992951 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 70864.4us
I20250116 02:59:04.003021 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:03.974300 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 46439.5us
I20250116 02:59:03.879853 22400 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:03.993178 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:04.009644 22374 raft_consensus.cc:2758] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250116 02:59:04.065856 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:04.082520 22379 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.973969 22126 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:04.009933 20676 raft_consensus.cc:2388] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 2: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:04.147480 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 144519us
I20250116 02:59:04.159010 20571 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:04.161166 20571 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:03.931528 22347 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:04.068107 22365 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:04.067615 22391 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:03.995780 22334 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.919864 22405 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:03.922344 22352 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:04.160449 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48834 (request call id 15) took 1856 ms. Trace:
I20250116 02:59:04.165030 22126 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:04.176862 20601 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:03.877835 22394 consensus_queue.cc:260] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:04.064828 22276 consensus_queue.cc:260] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:04.198387 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.032s sys 0.030s
W20250116 02:59:04.199189 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.032s sys 0.031s
W20250116 02:59:04.199988 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 279604us
I20250116 02:59:04.200829 20723 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:04.202107 20723 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:03.666220 22329 consensus_queue.cc:1159] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250116 02:59:04.195785 20680 rpcz_store.cc:276] 0116 02:59:02.297449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.297652 (+ 203us) service_pool.cc:225] Handling call
0116 02:59:04.009773 (+1712121us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54002e0ca0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326ca5020 00007fc326d099c2 00007fc32bc7082a 00007fc325349d14 00007fc325349ca2 00007fc325349bd1 00007fc325349b4c 00007fc325348e02 00007fc32513adfc 00007fc32513a236
0116 02:59:04.153928 (+144155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":841961728}
I20250116 02:59:04.210213 20723 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:04.193526 22379 raft_consensus.cc:2798] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:04.222363 20723 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:04.212167 22347 raft_consensus.cc:2798] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:04.223574 22126 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:04.232971 22384 consensus_meta.cc:287] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.630s user 0.010s sys 0.004s
I20250116 02:59:04.238302 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 268) took 1610 ms. Trace:
I20250116 02:59:04.229367 22276 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:04.224229 22379 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:04.231433 22347 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:04.240499 22407 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:04.242697 22126 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250116 02:59:04.237599 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 76531.4us
I20250116 02:59:04.222141 22352 consensus_queue.cc:1046] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250116 02:59:04.227361 22405 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:04.238891 22345 raft_consensus.cc:3049] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
W20250116 02:59:04.307734 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 428786us
W20250116 02:59:04.346824 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 360807us
I20250116 02:59:04.346850 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:04.347620 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.311738 20571 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:04.373672 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 62042.3us
I20250116 02:59:04.374712 20571 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:04.349126 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:04.394749 20571 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:04.303059 22379 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:59:04.456332 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 61736.9us
I20250116 02:59:04.305848 22292 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:04.485469 20753 raft_consensus.cc:1235] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:04.307539 22347 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:04.479494 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.485329 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.480188 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.496213 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.487119 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.498811 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.500559 22408 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:04.501140 22406 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:04.502012 20721 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.033s sys 0.070s
I20250116 02:59:04.258016 20754 rpcz_store.cc:276] 0116 02:59:02.628052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.635897 (+ 7845us) service_pool.cc:225] Handling call
0116 02:59:02.655218 (+ 19321us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:02.798692 (+143474us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:02.798747 (+ 55us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:02.798823 (+ 76us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:02.798869 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:04.237990 (+1439121us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54001d1854. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:04.238060 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:04.238215 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293419392}
W20250116 02:59:04.512459 20721 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.830s user 0.034s sys 0.070s
I20250116 02:59:04.518352 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.512076 20571 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:04.517118 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.457013 22333 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:04.259445 22384 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:04.546129 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.452384 22374 raft_consensus.cc:2758] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250116 02:59:04.482959 22352 consensus_queue.cc:1225] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer 231d91433a404468a44ea0a107345aeb log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250116 02:59:04.489109 22365 raft_consensus.cc:2758] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250116 02:59:04.479928 22345 raft_consensus.cc:738] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:04.553706 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.563647 20757 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:04.512724 22402 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:04.513316 20649 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.012s sys 0.021s
W20250116 02:59:04.581521 20649 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.660s user 0.013s sys 0.022s
W20250116 02:59:04.581907 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 660391us
I20250116 02:59:04.602648 22348 raft_consensus.cc:2758] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250116 02:59:04.497763 22292 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:04.578323 22390 consensus_meta.cc:287] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.860s user 0.014s sys 0.003s
W20250116 02:59:04.650095 22387 consensus_meta.cc:287] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.988s user 0.008s sys 0.004s
I20250116 02:59:04.623878 22281 raft_consensus.cc:3049] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
W20250116 02:59:04.745194 20722 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.073s sys 0.121s
W20250116 02:59:04.755721 20722 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.600s user 0.074s sys 0.121s
I20250116 02:59:04.718690 22281 raft_consensus.cc:738] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:04.501294 22408 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:04.650718 22390 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:04.789522 22404 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:04.791501 22404 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:04.629612 22402 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:04.720880 22387 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:04.774183 20572 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.007s sys 0.014s
I20250116 02:59:04.746399 22345 consensus_queue.cc:260] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.401726 22390 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:05.311138 21299 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.317s user 0.067s sys 0.055s
W20250116 02:59:05.490309 21299 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.496s user 0.069s sys 0.056s
I20250116 02:59:04.879768 22384 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:05.452762 22388 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:04.793859 22309 consensus_meta.cc:287] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.187s user 0.008s sys 0.006s
I20250116 02:59:04.797781 22281 consensus_queue.cc:260] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:05.475209 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:04.797114 22292 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:05.477569 22387 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:05.478169 20572 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.332s user 0.008s sys 0.020s
W20250116 02:59:05.291893 22260 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.002s sys 0.011s
W20250116 02:59:04.876585 21328 consensus_meta.cc:287] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.284s user 0.014s sys 0.000s
W20250116 02:59:05.352984 22333 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.000s sys 0.011s
I20250116 02:59:04.797247 22404 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:05.455866 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 943827us
W20250116 02:59:05.270161 20757 consensus_meta.cc:287] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.697s user 0.009s sys 0.000s
I20250116 02:59:04.757242 22399 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:05.482818 20749 consensus_meta.cc:287] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.115s user 0.020s sys 0.000s
W20250116 02:59:05.491035 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 1.49687e+06us
I20250116 02:59:05.473866 22345 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:59:05.577178 20608 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.434s user 0.006s sys 0.016s
I20250116 02:59:05.541988 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:05.719585 20608 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.576s user 0.006s sys 0.017s
W20250116 02:59:04.817162 22411 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20250116 02:59:05.580777 20755 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:05.273458 22393 consensus_meta.cc:287] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 1.475s user 0.005s sys 0.009s
W20250116 02:59:04.849287 22400 consensus_meta.cc:287] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.785s user 0.007s sys 0.001s
W20250116 02:59:05.587723 20572 connection.cc:232] RPC call timeout handler was delayed by -1.3772213869999632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21530000000000002s for a response
W20250116 02:59:04.826539 22366 consensus_meta.cc:287] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.280s user 0.008s sys 0.005s
I20250116 02:59:04.848938 22347 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:05.348228 22373 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.044s user 0.004s sys 0.010s
I20250116 02:59:04.850428 22416 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:05.610379 22430 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:05.451719 22375 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.117s user 0.002s sys 0.024s
I20250116 02:59:05.626382 21328 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:04.854957 22420 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:05.471182 22335 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:05.628350 20723 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.378s user 0.001s sys 0.026s
I20250116 02:59:05.468787 22405 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:05.493355 22351 raft_consensus.cc:3049] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:04.832648 22408 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:05.716126 20749 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:05.672334 20757 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:05.821627 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1746) took 1943 ms. Trace:
I20250116 02:59:05.718207 20571 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:05.729666 20648 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.098s user 0.013s sys 0.067s
W20250116 02:59:05.626883 22276 consensus_meta.cc:287] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.322s user 0.010s sys 0.003s
W20250116 02:59:05.625835 22333 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.996s user 0.000s sys 0.013s
W20250116 02:59:05.605304 22260 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.268s user 0.003s sys 0.013s
W20250116 02:59:05.631484 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.887s user 0.002s sys 0.015s
W20250116 02:59:05.630946 20646 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.262s user 0.015s sys 0.004s
W20250116 02:59:05.631186 20756 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.026s user 0.006s sys 0.012s
W20250116 02:59:05.631275 20647 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.967s user 0.004s sys 0.014s
I20250116 02:59:05.594170 22402 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:05.762274 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:05.715227 22281 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:05.756765 20754 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:05.721649 22387 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:05.820971 20723 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.570s user 0.006s sys 0.027s
I20250116 02:59:05.605751 22309 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.721153 22424 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:05.784786 22373 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.481s user 0.006s sys 0.010s
I20250116 02:59:05.626072 22388 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.772753 22292 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:05.827483 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.091s user 0.004s sys 0.019s
I20250116 02:59:05.798401 22384 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:05.822978 20574 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.010s sys 0.021s
I20250116 02:59:05.763270 22418 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:05.827570 20646 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.458s user 0.018s sys 0.007s
I20250116 02:59:05.827903 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1749) took 1346 ms. Trace:
W20250116 02:59:05.831044 20756 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.226s user 0.006s sys 0.018s
W20250116 02:59:05.830051 20648 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.658s user 0.014s sys 0.068s
I20250116 02:59:05.824754 20608 rpcz_store.cc:276] 0116 02:59:03.878477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.878718 (+ 241us) service_pool.cc:225] Handling call
0116 02:59:03.933978 (+ 55260us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:03.934212 (+ 234us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:05.721424 (+1787212us) log.cc:844] Serialized 14 byte log entry
0116 02:59:05.820311 (+ 98887us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:05.821243 (+ 932us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:05.821302 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:05.821388 (+ 86us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:05.821549 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1675486,"threads_started":2}
W20250116 02:59:05.831151 20647 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.167s user 0.007s sys 0.016s
I20250116 02:59:05.799032 22390 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:05.773092 22426 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:05.820385 22399 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:05.830322 20749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1741) took 1477 ms (client timeout 1775 ms). Trace:
I20250116 02:59:05.756315 22393 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:05.828334 20755 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:05.826857 22375 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.609s user 0.003s sys 0.035s
W20250116 02:59:05.851240 20574 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.680s user 0.010s sys 0.021s
I20250116 02:59:05.833667 22408 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:05.824007 22351 raft_consensus.cc:738] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:05.826257 22405 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.839947 22402 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:05.832556 21328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 610) took 2347 ms (client timeout 2025 ms). Trace:
I20250116 02:59:05.853644 20757 rpcz_store.cc:276] 0116 02:59:04.478393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:04.499673 (+ 21280us) service_pool.cc:225] Handling call
0116 02:59:05.824582 (+1324909us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:05.882287 20755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1755) took 1357 ms (client timeout 1624 ms). Trace:
W20250116 02:59:05.876587 20749 rpcz_store.cc:269] 0116 02:59:04.347184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:04.347371 (+ 187us) service_pool.cc:225] Handling call
0116 02:59:05.824276 (+1476905us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:05.883051 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.013s sys 0.016s
W20250116 02:59:05.905536 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.822s user 0.013s sys 0.016s
W20250116 02:59:05.907689 21299 leader_election.cc:336] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 1.907s (SENT)
I20250116 02:59:05.833403 22335 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.774597 22366 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.911551 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:05.917333 20749 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:05.913527 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:05.882714 21327 raft_consensus.cc:1235] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:05.844445 22435 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:05.905313 20755 rpcz_store.cc:269] 0116 02:59:04.520137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:05.474863 (+954726us) service_pool.cc:225] Handling call
0116 02:59:05.877130 (+402267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:05.822342 22430 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.844671 22387 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:05.852269 22292 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.946122 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:05.971511 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 63874.7us
I20250116 02:59:05.871670 22384 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.831734 22420 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.876680 22390 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:05.981550 20753 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:05.887697 22408 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.985561 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1754) took 1465 ms. Trace:
I20250116 02:59:05.990842 20755 rpcz_store.cc:276] 0116 02:59:04.519864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:05.984644 (+1464780us) service_pool.cc:225] Handling call
0116 02:59:05.985033 (+ 389us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:05.985219 (+ 186us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:05.985245 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:05.985289 (+ 44us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:05.985320 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:05.985372 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:05.985493 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:05.992970 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1758) took 1453 ms. Trace:
I20250116 02:59:05.993345 20755 rpcz_store.cc:276] 0116 02:59:04.540004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:05.992008 (+1452004us) service_pool.cc:225] Handling call
0116 02:59:05.992416 (+ 408us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:05.992636 (+ 220us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:05.992662 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:05.992706 (+ 44us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:05.992745 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:05.992796 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:05.992911 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:05.833861 22416 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.924221 22347 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:05.987818 22430 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:05.997220 22422 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:05.909922 22402 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:05.903458 21328 rpcz_store.cc:269] 0116 02:59:03.475976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.478822 (+ 2846us) service_pool.cc:225] Handling call
0116 02:59:05.821563 (+2342741us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:06.006951 22405 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:05.882472 22418 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:06.008613 22388 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.010166 22390 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:06.011161 22309 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:06.011662 22292 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:05.995177 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1759) took 1371 ms. Trace:
I20250116 02:59:05.880329 22426 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:06.019980 22431 log.cc:926] Time spent T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20250116 02:59:06.011535 22364 raft_consensus.cc:3049] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:06.029398 20755 rpcz_store.cc:276] 0116 02:59:04.623489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:05.994347 (+1370858us) service_pool.cc:225] Handling call
0116 02:59:05.994680 (+ 333us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:05.994872 (+ 192us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:05.994897 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:05.994941 (+ 44us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:05.994971 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:05.995019 (+ 48us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:05.995121 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:05.847327 22424 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:06.013809 20748 consensus_meta.cc:287] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.858s user 0.012s sys 0.004s
I20250116 02:59:06.016677 22347 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:06.012727 22430 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:05.882112 22399 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:05.945412 22435 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.043285 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1760) took 1315 ms. Trace:
I20250116 02:59:06.044377 22426 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.022110 22422 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:06.031945 22364 raft_consensus.cc:738] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:06.032364 20748 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:06.032033 22390 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:06.048046 20755 rpcz_store.cc:276] 0116 02:59:04.727495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:06.042398 (+1314903us) service_pool.cc:225] Handling call
0116 02:59:06.042781 (+ 383us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:06.042958 (+ 177us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:06.042993 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:06.043031 (+ 38us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:06.043056 (+ 25us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:06.043102 (+ 46us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:06.043228 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:06.058827 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 234358us
I20250116 02:59:06.024190 22351 consensus_queue.cc:260] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:06.066020 22351 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:06.031191 22347 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.076695 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:06.080850 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:06.080358 20757 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:06.082937 20757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1756) took 1562 ms (client timeout 2057 ms). Trace:
W20250116 02:59:06.083300 20757 rpcz_store.cc:269] 0116 02:59:04.520374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:05.913176 (+1392802us) service_pool.cc:225] Handling call
0116 02:59:06.082876 (+169700us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:06.070817 20755 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:59:06.087141 20606 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:06.088126 22428 log.cc:926] Time spent T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20250116 02:59:06.088217 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1742) took 1740 ms. Trace:
I20250116 02:59:06.069079 22340 raft_consensus.cc:2238] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250116 02:59:06.040711 22430 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.086488 20749 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:06.088260 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:06.088670 20756 rpcz_store.cc:276] 0116 02:59:04.347465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:04.348089 (+ 624us) service_pool.cc:225] Handling call
0116 02:59:04.481059 (+132970us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:04.494988 (+ 13929us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:04.495025 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:04.515938 (+ 20913us) log.cc:844] Serialized 26 byte log entry
0116 02:59:05.853881 (+1337943us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:05.853923 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:05.854009 (+ 86us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:06.087991 (+233982us) raft_consensus.cc:1680] finished
0116 02:59:06.088017 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:06.088159 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35324,"raft.run_cpu_time_us":2444,"raft.run_wall_time_us":38976,"thread_start_us":1248664,"threads_started":1,"wal-append.queue_time_us":1285654}
I20250116 02:59:06.093044 22335 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.098268 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1761) took 1335 ms. Trace:
I20250116 02:59:06.099741 20749 rpcz_store.cc:276] 0116 02:59:04.762892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:05.911233 (+1148341us) service_pool.cc:225] Handling call
0116 02:59:06.098203 (+186970us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:06.101275 20752 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:06.077018 22393 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:06.079411 20748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1381) took 2265 ms (client timeout 2153 ms). Trace:
W20250116 02:59:06.110054 22425 log.cc:926] Time spent T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20250116 02:59:06.112685 22435 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:06.069108 22390 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.115250 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:06.117681 21301 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.019s user 0.069s sys 0.116s
I20250116 02:59:06.124336 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:06.125051 22436 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:06.125231 20606 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:06.106750 22384 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.129706 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:06.130429 21301 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.090s user 0.070s sys 0.116s
W20250116 02:59:06.120827 20572 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.153s (SENT) [suppressed 39 similar messages]
W20250116 02:59:06.107371 20748 rpcz_store.cc:269] 0116 02:59:03.802273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.811593 (+ 9320us) service_pool.cc:225] Handling call
0116 02:59:06.068021 (+2256428us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:06.141288 22429 log.cc:926] Time spent T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20250116 02:59:06.119743 22387 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:06.144434 22432 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:06.141151 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:06.123250 22408 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:06.120359 22437 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:06.119426 22416 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.133775 22402 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:06.165620 22424 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:06.145398 22340 raft_consensus.cc:2267] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250116 02:59:06.108328 22364 consensus_queue.cc:260] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.124862 22418 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:06.146513 22432 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:06.162989 22379 consensus_meta.cc:287] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 1.669s user 0.015s sys 0.000s
I20250116 02:59:06.175256 22443 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:06.145030 22436 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.187995 22399 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.193631 22340 ts_tablet_manager.cc:1905] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250116 02:59:06.194454 22347 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:06.186890 22364 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:06.214193 20753 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:06.178683 22422 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:06.188594 22430 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:06.183359 22379 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:06.227191 22436 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.218551 22347 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:06.286691 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 446700us
I20250116 02:59:06.176497 22437 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.465209 22437 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:06.468549 20723 leader_election.cc:336] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.557s (SENT)
I20250116 02:59:06.199811 22443 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:06.427631 22333 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.003s sys 0.023s
I20250116 02:59:06.289275 22422 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:06.400986 22402 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:06.399853 22347 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:06.645069 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.000s sys 0.039s
W20250116 02:59:06.645723 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.727s user 0.000s sys 0.040s
W20250116 02:59:06.632155 22345 consensus_meta.cc:287] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.903s user 0.013s sys 0.001s
I20250116 02:59:06.489560 22430 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:06.490417 22387 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:06.490981 22416 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:06.493286 22432 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:06.643747 22333 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s user 0.003s sys 0.025s
I20250116 02:59:06.494410 22439 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:06.492184 22243 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:06.648687 22375 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.012s sys 0.029s
I20250116 02:59:06.648054 22390 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.675221 22430 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.686558 22430 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:06.687146 22430 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:06.687593 22430 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.688361 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 275) took 3255 ms. Trace:
I20250116 02:59:06.688885 20750 rpcz_store.cc:276] 0116 02:59:03.432651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.432763 (+ 112us) service_pool.cc:225] Handling call
0116 02:59:03.433131 (+ 368us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:03.436562 (+ 3431us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:03.436594 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:03.436640 (+ 46us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:03.436670 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:06.687953 (+3251283us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b540007cbd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:06.688041 (+ 88us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:06.688239 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7419264}
I20250116 02:59:06.679363 22432 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:06.690501 22432 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.693090 22432 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:06.693707 22432 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:06.563588 22443 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:06.694298 22432 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.800197 22432 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:06.647905 22422 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:06.718888 22375 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.770s user 0.014s sys 0.030s
I20250116 02:59:06.680909 22450 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:06.689582 22430 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:06.651641 22402 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.674293 22387 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:06.678562 22416 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.719281 22243 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.742234 22439 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.807000 22450 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:06.850960 22402 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.851621 22439 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:06.858356 22416 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:06.858737 22387 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:06.860249 22351 consensus_meta.cc:287] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.794s user 0.002s sys 0.011s
I20250116 02:59:06.865408 22450 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:06.866205 22422 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:07.382195 20647 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.504s user 0.013s sys 0.045s
W20250116 02:59:07.383203 20647 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.505s user 0.014s sys 0.045s
W20250116 02:59:07.528666 20647 connection.cc:232] RPC call timeout handler was delayed by -3.1315725719999818s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1714s for a response
W20250116 02:59:07.529278 20647 connection.cc:232] RPC call timeout handler was delayed by -3.0766725720000068s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
W20250116 02:59:07.529712 20647 connection.cc:232] RPC call timeout handler was delayed by -3.0667725719999908s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1786s for a response
W20250116 02:59:07.530058 20647 connection.cc:232] RPC call timeout handler was delayed by -3.0532725719999689s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20250116 02:59:07.530370 20647 connection.cc:232] RPC call timeout handler was delayed by -2.9524725719999196s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20250116 02:59:07.530738 20647 connection.cc:232] RPC call timeout handler was delayed by -2.9146725719999722s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20250116 02:59:07.531239 20647 connection.cc:232] RPC call timeout handler was delayed by -2.8642725720000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20110000000000003s for a response
W20250116 02:59:07.605834 20647 connection.cc:232] RPC call timeout handler was delayed by -2.8336725719999549s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20450000000000002s for a response
W20250116 02:59:07.606884 20647 connection.cc:232] RPC call timeout handler was delayed by -2.7499725719999333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
W20250116 02:59:07.607350 20647 connection.cc:232] RPC call timeout handler was delayed by -2.7027831099999275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16240000000000002s for a response
W20250116 02:59:07.608018 20647 connection.cc:232] RPC call timeout handler was delayed by -2.6959725719999597s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
W20250116 02:59:07.608398 20647 connection.cc:232] RPC call timeout handler was delayed by -2.6797725719999335s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22160000000000002s for a response
W20250116 02:59:07.608731 20647 connection.cc:232] RPC call timeout handler was delayed by -2.6316831099999263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
W20250116 02:59:07.609082 20647 connection.cc:232] RPC call timeout handler was delayed by -2.540783109999893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
W20250116 02:59:07.610129 20647 connection.cc:232] RPC call timeout handler was delayed by -2.4462831099998539s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20250116 02:59:07.972103 20647 connection.cc:232] RPC call timeout handler was delayed by -2.4330224859999134s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
W20250116 02:59:07.972743 20647 connection.cc:232] RPC call timeout handler was delayed by -2.3130831099998659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
W20250116 02:59:07.973398 20647 connection.cc:232] RPC call timeout handler was delayed by -2.1409129809999285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2025s for a response
W20250116 02:59:08.249708 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 273946us
I20250116 02:59:08.250932 20647 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:08.252976 20647 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:08.468655 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 215762us
I20250116 02:59:08.469882 20647 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:08.406909 20720 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.689s user 0.041s sys 0.055s
W20250116 02:59:08.518914 20720 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.801s user 0.041s sys 0.056s
I20250116 02:59:08.508654 22243 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:08.525763 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 55997.6us
I20250116 02:59:08.527259 20647 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:08.529402 22466 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:08.530305 22466 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:08.530968 22466 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:08.531697 22466 raft_consensus.cc:2743] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250116 02:59:08.535140 21301 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.027s user 0.041s sys 0.071s
W20250116 02:59:08.535723 21301 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.028s user 0.041s sys 0.072s
I20250116 02:59:08.536584 22469 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:08.537233 22469 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.538640 22408 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:08.539238 22408 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.540222 22469 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.540819 22469 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.541385 22408 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:08.541270 22469 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.543355 22469 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:08.544328 20647 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:08.545352 22467 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:08.535043 22468 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.549643 22468 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.553602 22326 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:08.553978 20720 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:08.554224 22470 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.554859 22326 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.553822 22468 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.556255 22469 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:08.554983 22470 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.557608 22326 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.558952 22470 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:08.700440 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 145025us
I20250116 02:59:08.702031 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:08.702257 20720 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:08.702910 20606 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:59:08.703281 22472 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:08.704298 20606 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1494) took 2619 ms (client timeout 1882 ms). Trace:
W20250116 02:59:08.704707 20606 rpcz_store.cc:269] 0116 02:59:06.084807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:06.144206 (+ 59399us) service_pool.cc:225] Handling call
0116 02:59:08.704191 (+2559985us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:08.705801 20606 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1505) took 2609 ms (client timeout 2054 ms). Trace:
W20250116 02:59:08.706251 20606 rpcz_store.cc:269] 0116 02:59:06.109617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.705600 (+2595983us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:08.705716 (+ 116us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:08.708034 20606 raft_consensus.cc:1235] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20250116 02:59:08.709086 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1495) took 2624 ms. Trace:
I20250116 02:59:08.709518 20606 rpcz_store.cc:276] 0116 02:59:06.086808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.706968 (+2620160us) service_pool.cc:225] Handling call
0116 02:59:08.707620 (+ 652us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:08.708721 (+ 1101us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.709031 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:08.711237 20606 raft_consensus.cc:1235] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:08.711995 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1496) took 2624 ms. Trace:
I20250116 02:59:08.712721 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:08.712953 20606 rpcz_store.cc:276] 0116 02:59:06.094259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.710594 (+2616335us) service_pool.cc:225] Handling call
0116 02:59:08.711037 (+ 443us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:08.711764 (+ 727us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.711946 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:08.715262 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:08.716365 20720 connection.cc:232] RPC call timeout handler was delayed by -1.3614442409999583s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2054s for a response
I20250116 02:59:08.716620 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:08.716775 22392 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.717239 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:08.717759 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.718132 21334 raft_consensus.cc:2388] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 2: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:08.718458 20750 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:08.719126 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:08.719511 22470 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:08.719820 20750 raft_consensus.cc:1270] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:59:08.720234 21334 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:59:08.720082 22470 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.720592 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.721166 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:08.717391 22392 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.722415 22470 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:08.723073 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:08.719034 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.721235 20757 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:08.724056 21334 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:08.724181 20720 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:08.724932 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.724987 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:08.725565 20757 raft_consensus.cc:2371] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3. Re-sending same reply.
I20250116 02:59:08.725023 22354 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.725126 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.727422 20748 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:08.727347 22354 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.728192 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.728585 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.729278 20748 raft_consensus.cc:2371] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2. Re-sending same reply.
I20250116 02:59:08.729363 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.723513 22392 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.729794 22354 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:08.717312 21330 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:08.731212 20647 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:08.726462 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.732136 22369 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:08.732679 20647 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:08.727048 22470 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:08.730329 22392 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:08.733008 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.734027 22470 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:08.730270 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.732626 22369 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:08.733093 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.735613 22369 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:08.734490 20750 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:08.736449 20748 raft_consensus.cc:2388] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:08.736634 20720 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:08.734511 20574 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.855s user 0.005s sys 0.074s
I20250116 02:59:08.734583 20647 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:08.737561 22326 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:08.738019 22326 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:08.738385 22326 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:08.734696 22470 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:08.737478 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.739846 20748 raft_consensus.cc:2388] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 2: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:08.737576 20574 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.858s user 0.006s sys 0.074s
I20250116 02:59:08.740963 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.742908 22466 raft_consensus.cc:2798] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:08.738533 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.748337 20757 raft_consensus.cc:2371] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3. Re-sending same reply.
I20250116 02:59:08.747917 22470 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.747371 20748 raft_consensus.cc:2371] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3. Re-sending same reply.
I20250116 02:59:08.747812 22466 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:08.749219 20647 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:08.750211 22466 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:08.750681 22470 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.751194 22354 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.751394 22255 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:59:08.751856 22354 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.752163 22474 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:08.753031 22474 raft_consensus.cc:695] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:08.753064 22255 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:08.754498 22474 consensus_queue.cc:237] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.734906 22392 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.756415 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.757212 20748 raft_consensus.cc:2371] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2. Re-sending same reply.
I20250116 02:59:08.757623 22392 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.759073 20606 raft_consensus.cc:1235] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250116 02:59:08.759353 22264 raft_consensus.cc:3049] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:08.759779 22264 raft_consensus.cc:738] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:08.760634 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1497) took 2666 ms. Trace:
I20250116 02:59:08.760711 22264 consensus_queue.cc:260] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.761159 22369 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.749372 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.762784 22264 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:08.763092 20757 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:08.763669 20647 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:08.764364 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.766122 22267 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:08.766652 22267 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:08.767022 22267 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:08.758332 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.779620 20647 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:08.758352 22392 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:08.781265 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.785169 20748 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:08.780061 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.787058 20757 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:08.785980 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:08.787894 21330 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:08.786852 20647 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:08.760999 20606 rpcz_store.cc:276] 0116 02:59:06.094586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.713777 (+2619191us) service_pool.cc:225] Handling call
0116 02:59:08.758810 (+ 45033us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:08.760306 (+ 1496us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.760563 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:08.792806 20720 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:08.783619 22392 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.783424 22255 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:08.793774 22255 raft_consensus.cc:695] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:08.794252 22354 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:08.794835 22255 consensus_queue.cc:237] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.795590 20647 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:08.793215 20572 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 2.657s user 0.028s sys 0.064s
W20250116 02:59:08.798177 20572 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.662s user 0.029s sys 0.065s
W20250116 02:59:08.798465 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 2.6774e+06us
I20250116 02:59:08.800310 22392 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.800787 22392 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:08.800619 22260 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:08.801337 22392 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.801261 22324 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:08.802559 22324 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:08.794862 22354 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:08.803150 22324 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:08.804709 22260 raft_consensus.cc:2798] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:08.803297 22354 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.805207 22260 raft_consensus.cc:695] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:08.805056 21330 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:08.806188 22260 consensus_queue.cc:237] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.811167 20720 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:08.811993 22345 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.813753 22473 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.812315 22470 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:08.815856 22470 raft_consensus.cc:695] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:08.816949 22470 consensus_queue.cc:237] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.817238 22326 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.812440 22354 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.823482 22354 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:08.824262 22354 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.826059 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1365) took 6877 ms. Trace:
I20250116 02:59:08.827690 22473 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.828194 20681 rpcz_store.cc:276] 0116 02:59:01.948147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.278471 (+330324us) service_pool.cc:225] Handling call
0116 02:59:02.881213 (+602742us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:02.881518 (+ 305us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:03.713188 (+831670us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.825594 (+5112406us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b5400319cd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:08.825904 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1895121536}
I20250116 02:59:08.829959 22473 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:08.830446 20754 consensus_meta.cc:287] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 2.974s user 0.014s sys 0.037s
I20250116 02:59:08.831086 20754 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:08.831779 22400 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.815876 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1498) took 2720 ms. Trace:
I20250116 02:59:08.812934 20647 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:08.786170 22267 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.796876 22369 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.814095 22345 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.761776 22466 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.834110 20606 rpcz_store.cc:276] 0116 02:59:06.094857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.793140 (+2698283us) service_pool.cc:225] Handling call
0116 02:59:08.815200 (+ 22060us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:08.815405 (+ 205us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:08.815430 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:08.815472 (+ 42us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:08.815501 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.815550 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:08.815691 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:08.856793 20572 connection.cc:232] RPC call timeout handler was delayed by -1.8063290140000845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
W20250116 02:59:08.857378 20572 connection.cc:232] RPC call timeout handler was delayed by -1.422929014000033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
W20250116 02:59:08.857717 20572 connection.cc:232] RPC call timeout handler was delayed by -1.3084551390001025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20250116 02:59:08.858034 20572 connection.cc:232] RPC call timeout handler was delayed by -1.1518551390000766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
I20250116 02:59:08.859689 20572 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:08.840682 20750 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:08.827203 22354 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.830538 22473 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.849949 22330 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:08.860940 20572 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:08.849123 20754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1757) took 4321 ms (client timeout 1703 ms). Trace:
I20250116 02:59:08.832492 22400 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.864836 20572 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:08.866567 20572 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:08.863895 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.867185 20606 raft_consensus.cc:1235] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:08.867591 20750 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:08.868142 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1500) took 2772 ms. Trace:
I20250116 02:59:08.869776 20606 rpcz_store.cc:276] 0116 02:59:06.095313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.860758 (+2765445us) service_pool.cc:225] Handling call
0116 02:59:08.867016 (+ 6258us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:08.867902 (+ 886us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.868083 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:08.871397 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.872349 20750 raft_consensus.cc:2388] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:08.874418 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.876112 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.876838 20750 raft_consensus.cc:2371] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3. Re-sending same reply.
I20250116 02:59:08.877808 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.878926 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.879088 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.879736 20750 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:08.879678 20756 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:08.881065 20756 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:08.882323 20756 raft_consensus.cc:1235] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:08.882364 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 242) took 2816 ms. Trace:
I20250116 02:59:08.883433 20608 rpcz_store.cc:276] 0116 02:59:06.065649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:06.066668 (+ 1019us) service_pool.cc:225] Handling call
0116 02:59:06.081939 (+ 15271us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:06.135211 (+ 53272us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:06.135247 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:06.135298 (+ 51us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:06.135329 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.882129 (+2746800us) spinlock_profiling.cc:243] Waited 53.1 ms on lock 0x7b5400091d54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:08.882190 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:08.882306 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122026240}
I20250116 02:59:08.884007 20647 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
W20250116 02:59:08.863608 20754 rpcz_store.cc:269] 0116 02:59:04.520625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:05.467208 (+946583us) service_pool.cc:225] Handling call
0116 02:59:08.842382 (+3375174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:08.868911 20748 raft_consensus.cc:1235] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:08.871985 22477 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:08.886468 22477 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:08.886961 22477 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:08.889922 20756 raft_consensus.cc:1235] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20250116 02:59:08.891448 22345 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.892120 22345 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:08.892460 22345 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:08.892802 22345 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:08.867050 22354 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:08.893831 22354 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:08.894302 22354 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:08.885695 22330 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:08.894683 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:08.896255 22466 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:08.897346 22324 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.898367 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:08.899049 20756 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:08.899682 22324 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.900168 22330 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:08.900600 22330 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:08.867666 22473 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
W20250116 02:59:08.899827 20601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1485) took 5263 ms (client timeout 1609 ms). Trace:
I20250116 02:59:08.899852 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1501) took 2804 ms. Trace:
I20250116 02:59:08.904254 20606 rpcz_store.cc:276] 0116 02:59:06.095535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.870884 (+2775349us) service_pool.cc:225] Handling call
0116 02:59:08.899345 (+ 28461us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:08.899541 (+ 196us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:08.899566 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:08.899612 (+ 46us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:08.899641 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.899693 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:08.899793 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:08.900753 20608 raft_consensus.cc:1235] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:08.907451 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1502) took 2811 ms. Trace:
I20250116 02:59:08.907707 20608 rpcz_store.cc:276] 0116 02:59:06.095744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.884500 (+2788756us) service_pool.cc:225] Handling call
0116 02:59:08.900567 (+ 16067us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:08.907198 (+ 6631us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.907399 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:08.902993 20601 rpcz_store.cc:269] 0116 02:59:03.635865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.635973 (+ 108us) service_pool.cc:225] Handling call
0116 02:59:08.899591 (+5263618us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b54000573d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:08.899722 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1194431872}
I20250116 02:59:08.910658 20601 raft_consensus.cc:1235] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:08.909346 20750 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:08.911998 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:08.912853 21331 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:08.917193 22476 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:08.917847 22476 raft_consensus.cc:695] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:08.914374 20647 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:08.922766 22473 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:08.923437 22473 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:08.923837 22267 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.918851 22476 consensus_queue.cc:237] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.926434 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1504) took 2830 ms. Trace:
W20250116 02:59:08.926764 20649 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 3.034s user 0.025s sys 0.056s
I20250116 02:59:08.926975 20608 rpcz_store.cc:276] 0116 02:59:06.096163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.908308 (+2812145us) service_pool.cc:225] Handling call
0116 02:59:08.925892 (+ 17584us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:08.926090 (+ 198us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:08.926116 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:08.926161 (+ 45us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:08.926189 (+ 28us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.926241 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:08.926367 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:08.928257 20649 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 3.036s user 0.025s sys 0.056s
I20250116 02:59:08.929116 22415 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.914808 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1503) took 2818 ms. Trace:
I20250116 02:59:08.930202 20606 rpcz_store.cc:276] 0116 02:59:06.095957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.905482 (+2809525us) service_pool.cc:225] Handling call
0116 02:59:08.914325 (+ 8843us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:08.914498 (+ 173us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:08.914523 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:08.914567 (+ 44us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:08.914592 (+ 25us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.914667 (+ 75us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:08.914751 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:08.932756 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 66285.4us
I20250116 02:59:08.934274 22388 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:08.934998 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:08.929661 22415 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.939301 22342 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:08.939946 22342 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.924222 22298 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:08.941522 22298 raft_consensus.cc:695] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:08.942646 22298 consensus_queue.cc:237] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.944412 22342 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.946128 22415 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.941249 22477 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.952487 22330 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.954005 22345 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.955489 21331 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:08.956503 20720 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:08.956838 22345 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.957507 22419 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:08.958192 22419 raft_consensus.cc:695] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:08.959092 22419 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.960316 20756 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:08.962329 20647 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:08.963368 22267 raft_consensus.cc:2798] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:08.964679 22267 raft_consensus.cc:695] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:08.966118 22267 consensus_queue.cc:237] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.956068 22330 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:08.974498 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:08.956467 22392 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.923938 22473 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:08.973172 22354 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:08.983846 22354 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:08.984889 22473 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.989430 22477 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:08.998111 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:09.000923 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.003482 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 616) took 5276 ms. Trace:
W20250116 02:59:09.001006 20682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1383) took 6871 ms (client timeout 2042 ms). Trace:
W20250116 02:59:09.005220 20682 rpcz_store.cc:269] 0116 02:59:02.129756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.246480 (+116724us) service_pool.cc:225] Handling call
0116 02:59:09.000749 (+6754269us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54000866d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:09.000896 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1654197504}
I20250116 02:59:09.003939 21335 rpcz_store.cc:276] 0116 02:59:03.727389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.728473 (+ 1084us) service_pool.cc:225] Handling call
0116 02:59:03.744021 (+ 15548us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:03.786696 (+ 42675us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.003199 (+5216503us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x7b54004a06d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:09.003388 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96035584}
I20250116 02:59:09.003978 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:09.014525 20750 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.015441 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1714) took 6508 ms. Trace:
I20250116 02:59:09.029103 20750 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:09.030390 20647 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:09.031178 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:09.023418 20599 rpcz_store.cc:276] 0116 02:59:02.507464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.511209 (+ 3745us) service_pool.cc:225] Handling call
0116 02:59:03.012861 (+501652us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:03.013153 (+ 292us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:03.460841 (+447688us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.015107 (+5554266us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54002e6dd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:09.015358 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":771446912}
I20250116 02:59:09.030191 22420 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:09.043704 22267 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:09.047528 22267 raft_consensus.cc:695] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:09.048134 22340 ts_tablet_manager.cc:1918] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250116 02:59:09.048743 22340 log.cc:1198] T 38583cae882144f48cec12c72ba697cf P 17446c8dacf04461acb843547665250f: Deleting WAL directory at /tmp/dist-test-taskSG8bRf/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736996228253955-19386-0/minicluster-data/ts-2-root/wals/38583cae882144f48cec12c72ba697cf
I20250116 02:59:09.053191 20749 raft_consensus.cc:3049] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:09.053665 20749 raft_consensus.cc:738] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:09.054436 20749 consensus_queue.cc:260] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.055622 20749 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.048681 22267 consensus_queue.cc:237] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.060849 22379 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:09.054817 22340 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50634 (request call id 598) took 6910 ms. Trace:
I20250116 02:59:09.064909 22340 rpcz_store.cc:276] 0116 02:59:02.144309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.150766 (+ 6457us) service_pool.cc:225] Handling call
0116 02:59:02.500835 (+350069us) ts_tablet_manager.cc:1145] Deleting tablet 38583cae882144f48cec12c72ba697cf
0116 02:59:02.529089 (+ 28254us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0116 02:59:06.859831 (+4330742us) tablet_metadata.cc:641] Metadata flushed
0116 02:59:09.047792 (+2187961us) tablet_metadata.cc:641] Metadata flushed
0116 02:59:09.054736 (+ 6944us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3381002,"tablet-delete.queue_time_us":261258,"thread_start_us":104065,"threads_started":1}
W20250116 02:59:09.070205 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 2.60143e+06us
W20250116 02:59:09.071208 20723 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.579s (SENT)
I20250116 02:59:09.071631 20723 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:09.073110 20749 raft_consensus.cc:1270] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250116 02:59:09.074038 20749 pending_rounds.cc:77] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 1
I20250116 02:59:09.074275 20749 pending_rounds.cc:99] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250116 02:59:09.074525 20749 raft_consensus.cc:2883] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:09.076910 22306 consensus_queue.cc:1035] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:09.082510 22377 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:09.084597 22324 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:09.086293 22324 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.090286 22324 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:09.083050 20723 leader_election.cc:336] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.644s (SENT)
I20250116 02:59:09.090535 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:09.091742 21335 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:09.093581 20720 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:09.094688 20723 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:09.094853 22324 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:09.095510 22324 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:09.095708 22377 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:09.095888 22324 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:09.149526 22376 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:09.196488 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.550s user 0.017s sys 0.041s
W20250116 02:59:09.218689 21299 connection.cc:232] RPC call timeout handler was delayed by -1.8320296390000976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
I20250116 02:59:09.218961 22303 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:09.219278 21299 connection.cc:232] RPC call timeout handler was delayed by -1.7294296390000454s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19690000000000002s for a response
I20250116 02:59:09.219738 22395 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:09.219856 22303 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.220440 22395 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.223492 22303 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:09.220153 21299 connection.cc:232] RPC call timeout handler was delayed by -1.6869743560000643s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
W20250116 02:59:09.224589 21299 connection.cc:232] RPC call timeout handler was delayed by -1.2343206360000067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
W20250116 02:59:09.224927 21299 connection.cc:232] RPC call timeout handler was delayed by -1.2001978880000479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
I20250116 02:59:09.236505 22366 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:09.294543 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.648s user 0.017s sys 0.044s
I20250116 02:59:09.238329 22385 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:09.371037 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:09.376757 21302 connection.cc:232] RPC call timeout handler was delayed by -2.7918341410000949s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
W20250116 02:59:09.377413 21302 connection.cc:232] RPC call timeout handler was delayed by -2.7251380559999916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19690000000000002s for a response
W20250116 02:59:09.377744 21302 connection.cc:232] RPC call timeout handler was delayed by -1.8034286910001356s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17620000000000002s for a response
W20250116 02:59:09.384917 22492 log.cc:926] Time spent T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.291s user 0.001s sys 0.000s
I20250116 02:59:09.397734 22395 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:09.410259 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 311373us
I20250116 02:59:09.412751 20723 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:09.413550 22281 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:09.413237 22324 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.413694 22366 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:09.414288 22366 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:09.414129 22281 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.414793 22366 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:09.418780 22377 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:09.418900 22281 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:09.419564 22281 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:09.419390 22377 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.420078 22281 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.421454 22377 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:09.422188 22281 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:09.423998 20751 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:09.424603 20750 raft_consensus.cc:1270] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:59:09.425118 20674 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:09.431340 22336 consensus_queue.cc:1035] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:09.436983 21335 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:09.439234 22347 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:09.439960 20721 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 3.855s user 0.033s sys 0.035s
W20250116 02:59:09.440579 20721 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 3.863s user 0.034s sys 0.036s
I20250116 02:59:09.443173 20749 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250116 02:59:09.443398 20648 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 3.555s user 0.009s sys 0.052s
W20250116 02:59:09.443120 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.949s user 0.017s sys 0.027s
W20250116 02:59:09.444046 20648 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 3.556s user 0.010s sys 0.052s
W20250116 02:59:09.444270 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.952s user 0.017s sys 0.027s
I20250116 02:59:09.445155 22255 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:09.445732 22255 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.447966 22248 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:09.448542 22248 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.449576 22255 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:09.443120 22364 consensus_meta.cc:287] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 3.228s user 0.017s sys 0.015s
I20250116 02:59:09.447434 22336 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:09.451069 20679 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:09.450603 22377 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:09.451858 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 218) took 6146 ms. Trace:
I20250116 02:59:09.451987 22377 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.452382 20679 rpcz_store.cc:276] 0116 02:59:03.305603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.308856 (+ 3253us) service_pool.cc:225] Handling call
0116 02:59:09.450456 (+6141600us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:09.450793 (+ 337us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:09.451556 (+ 763us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.451791 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3623680}
I20250116 02:59:09.451421 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.450520 22336 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.450202 22281 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:09.455485 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.455366 22281 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.455754 22336 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:09.455847 20574 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.010s sys 0.014s
W20250116 02:59:09.456545 20574 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.587s user 0.010s sys 0.014s
I20250116 02:59:09.457473 22281 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:08.938314 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:09.459368 22375 consensus_queue.cc:1046] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:59:09.460000 20608 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:59:09.460491 22377 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:09.461089 22375 consensus_queue.cc:1046] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250116 02:59:09.462126 22226 raft_consensus.cc:3049] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:09.462292 22375 raft_consensus.cc:3049] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:09.462507 20599 raft_consensus.cc:1270] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:59:09.462849 22375 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:09.462496 20674 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:59:09.463816 22375 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.465286 22375 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.466048 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:09.469288 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:09.469761 22343 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250116 02:59:09.470438 22226 raft_consensus.cc:738] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:09.470983 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:09.473431 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:09.473145 22226 consensus_queue.cc:260] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.475438 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.471984 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:09.474421 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:09.478263 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.478927 20680 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:09.480700 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 221) took 6173 ms. Trace:
I20250116 02:59:09.482841 20674 rpcz_store.cc:276] 0116 02:59:03.306821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.308894 (+ 2073us) service_pool.cc:225] Handling call
0116 02:59:03.382123 (+ 73229us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:09.424614 (+6042491us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b540009e5a0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326c9d66d 00007fc326c8b2b0 00007fc326ceeaba 00007fc326d063af 00007fc326d07b4e 00007fc326d11c89 00007fc326d12556 00007fc326d0eedf 00007fc326d0e99e 00007fc32bc6d2f9
0116 02:59:09.463886 (+ 39272us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.480613 (+ 16727us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1247350016}
I20250116 02:59:09.480801 21300 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:09.483546 22248 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:09.486233 20608 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250116 02:59:09.487442 21333 raft_consensus.cc:1235] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:09.489666 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 388) took 5665 ms. Trace:
I20250116 02:59:09.490250 21333 rpcz_store.cc:276] 0116 02:59:03.823897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.824044 (+ 147us) service_pool.cc:225] Handling call
0116 02:59:09.487034 (+5662990us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:09.487243 (+ 209us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:09.489220 (+ 1977us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.489576 (+ 356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40130944}
I20250116 02:59:09.474370 22226 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.492242 20749 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:42186
W20250116 02:59:09.495604 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1752) took 4977 ms (client timeout 1804 ms). Trace:
I20250116 02:59:09.495708 22347 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250116 02:59:09.496054 20752 rpcz_store.cc:269] 0116 02:59:04.517954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:04.518088 (+ 134us) service_pool.cc:225] Handling call
0116 02:59:09.495369 (+4977281us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540033d1d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:09.495512 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3009156352}
I20250116 02:59:09.498106 20599 raft_consensus.cc:3049] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:09.498695 20599 raft_consensus.cc:738] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:09.499049 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:09.498257 22347 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:09.499835 21335 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:09.499567 20599 consensus_queue.cc:260] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.488754 22326 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:09.488159 20608 raft_consensus.cc:1270] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:59:09.501040 22326 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:09.501467 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.502108 20608 raft_consensus.cc:3049] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:09.502470 20608 raft_consensus.cc:738] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:09.501013 20599 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:09.503515 20608 consensus_queue.cc:260] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.447981 22407 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:09.505125 22407 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:09.504333 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.530012 20649 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:09.509528 22366 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:09.531548 20757 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:59:09.511083 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:09.513090 20599 raft_consensus.cc:1270] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:59:09.513686 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:09.533463 21329 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.534067 20599 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:59:09.536408 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:09.537245 21335 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:59:09.537297 21333 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.538161 22226 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250116 02:59:09.539906 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 612) took 5992 ms. Trace:
I20250116 02:59:09.501243 22415 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250116 02:59:09.534147 22407 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:09.542462 22326 consensus_queue.cc:1046] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:09.543244 22415 consensus_queue.cc:1035] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:09.543735 22325 raft_consensus.cc:3049] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:09.544143 22325 raft_consensus.cc:738] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:09.505156 20608 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.545928 20752 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20250116 02:59:09.546125 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:09.540403 21327 rpcz_store.cc:276] 0116 02:59:03.547123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.547789 (+ 666us) service_pool.cc:225] Handling call
0116 02:59:03.608875 (+ 61086us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:05.923256 (+2314381us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.539600 (+3616344us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b54006900d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:09.539841 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5196407040}
I20250116 02:59:09.547140 21328 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 4
I20250116 02:59:09.546257 20752 pending_rounds.cc:77] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 3
I20250116 02:59:09.548501 20752 pending_rounds.cc:99] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20250116 02:59:09.548816 20752 raft_consensus.cc:2911] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20250116 02:59:09.549232 20752 raft_consensus.cc:2883] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:09.493192 20749 raft_consensus.cc:604] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:09.550796 20749 raft_consensus.cc:612] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:09.551786 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.552356 20749 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:09.504844 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1736) took 6093 ms. Trace:
I20250116 02:59:09.555043 20606 raft_consensus.cc:2388] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 2: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:09.556200 22377 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:09.553664 20607 rpcz_store.cc:276] 0116 02:59:03.411772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.411935 (+ 163us) service_pool.cc:225] Handling call
0116 02:59:03.412398 (+ 463us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:03.433191 (+ 20793us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:03.433228 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:03.438790 (+ 5562us) log.cc:844] Serialized 26 byte log entry
0116 02:59:08.901894 (+5463104us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b18000fda80. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32698be4b 00007fc326991913 00007fc326991cc1 00007fc326ce345f
0116 02:59:08.933999 (+ 32105us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:08.934039 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:08.934124 (+ 85us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:09.504511 (+570387us) raft_consensus.cc:1680] finished
0116 02:59:09.504535 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:09.504768 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20380,"raft.queue_time_us":43913,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":8319,"spinlock_wait_cycles":34388864,"thread_start_us":31864,"threads_started":1,"wal-append.queue_time_us":5482794}
I20250116 02:59:09.545009 22325 consensus_queue.cc:260] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.561867 22325 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.532546 22329 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:09.554073 20608 raft_consensus.cc:1270] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:59:09.568576 20647 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:09.569800 22381 consensus_queue.cc:1046] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:09.570128 22343 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:59:09.571256 22306 raft_consensus.cc:2798] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:09.571691 22306 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:09.572064 22306 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.572417 22343 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:59:09.514978 22248 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250116 02:59:09.517486 20537 debug-util.cc:398] Leaking SignalData structure 0x7b080003df40 after lost signal to thread 19389
I20250116 02:59:09.521270 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.521522 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:09.521644 20646 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 3.633s user 0.042s sys 0.031s
I20250116 02:59:09.523052 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.575016 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.524418 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.524587 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.575884 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.576262 20676 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:09.575860 20674 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:09.524863 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.576690 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.577672 20678 raft_consensus.cc:2388] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:09.577908 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.526041 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:09.577098 20646 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 3.689s user 0.043s sys 0.032s
I20250116 02:59:09.578370 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:09.578761 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 3.69636e+06us
I20250116 02:59:09.579221 20678 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:09.579419 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.577203 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:09.580271 20751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1740) took 5233 ms (client timeout 2216 ms). Trace:
I20250116 02:59:09.580381 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.570775 22381 consensus_queue.cc:1225] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer 231d91433a404468a44ea0a107345aeb log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250116 02:59:09.580647 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.580885 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1742) took 5902 ms. Trace:
I20250116 02:59:09.576761 20679 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:09.578263 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.581597 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:09.581945 20751 rpcz_store.cc:269] 0116 02:59:04.346887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:04.347032 (+ 145us) service_pool.cc:225] Handling call
0116 02:59:09.579852 (+5232820us) spinlock_profiling.cc:243] Waited 4.92 s on lock 0x7b540012e154. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:09.580202 (+ 350us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11316742528}
I20250116 02:59:09.582052 20678 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:09.584223 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.585495 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.586228 20678 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:09.581282 20605 rpcz_store.cc:276] 0116 02:59:03.676632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.677047 (+ 415us) service_pool.cc:225] Handling call
0116 02:59:03.677660 (+ 613us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:03.708571 (+ 30911us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:03.708609 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:03.708655 (+ 46us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:03.708687 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.578370 (+5869683us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b5400108954. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:09.578431 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:09.578565 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70555008}
I20250116 02:59:09.550427 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.529184 20604 raft_consensus.cc:1235] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:09.587960 20601 raft_consensus.cc:1270] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:09.530766 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.589097 20680 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:09.590170 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:09.583865 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.582051 20646 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:09.591612 20677 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
W20250116 02:59:09.574802 20537 debug-util.cc:398] Leaking SignalData structure 0x7b080014e640 after lost signal to thread 20540
I20250116 02:59:09.592660 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.593168 22467 raft_consensus.cc:2798] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:09.593786 22467 raft_consensus.cc:695] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:09.594038 22248 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:09.595768 22248 consensus_queue.cc:1035] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:59:09.596364 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.597277 21329 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:09.587198 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.547737 22415 consensus_queue.cc:1035] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:59:09.591641 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.599742 22298 consensus_queue.cc:1046] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250116 02:59:09.600273 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.601054 20680 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:09.601182 22342 raft_consensus.cc:3049] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:09.601593 22342 raft_consensus.cc:738] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:09.601841 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.513226 22324 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:09.597652 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:09.602543 22342 consensus_queue.cc:260] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.604043 22342 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.604285 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.606748 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:09.607828 22400 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:09.608089 20680 raft_consensus.cc:2388] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:09.608667 22467 consensus_queue.cc:237] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.603209 20682 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:09.593940 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.550100 22336 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:59:09.625435 22453 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:09.617000 21332 raft_consensus.cc:3049] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:09.626128 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.626828 21332 raft_consensus.cc:738] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:09.627760 21332 consensus_queue.cc:260] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.628077 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.626085 22453 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.629686 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.630856 22453 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:09.631978 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:09.632843 20681 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:09.632891 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:09.635365 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.636219 20752 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:09.637127 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.637470 20682 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:09.629115 21332 raft_consensus.cc:3054] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.649091 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 282) took 4072 ms. Trace:
I20250116 02:59:09.649801 20755 rpcz_store.cc:276] 0116 02:59:05.577198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:06.098681 (+521483us) service_pool.cc:225] Handling call
0116 02:59:06.099114 (+ 433us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:06.463836 (+364722us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:06.463876 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:06.463926 (+ 50us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:06.463969 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.648602 (+3184633us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b5400370fd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:09.648663 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:09.648979 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":838064512}
I20250116 02:59:09.651865 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:06.122527 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:09.657274 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.658897 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.660022 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.660827 20755 raft_consensus.cc:2363] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4.
I20250116 02:59:09.661440 20677 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:09.662866 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:09.663457 20600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 247) took 3583 ms (client timeout 1762 ms). Trace:
I20250116 02:59:09.638011 20503 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41956 (request call id 334) took 7489 ms. Trace:
W20250116 02:59:09.664007 20600 rpcz_store.cc:269] 0116 02:59:06.080226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:06.088590 (+ 8364us) service_pool.cc:225] Handling call
0116 02:59:09.656167 (+3567577us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:09.633819 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.638695 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:09.667518 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.663429 20682 raft_consensus.cc:1235] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:09.668659 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.669315 20750 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:09.673553 20750 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:09.674521 20749 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:09.664528 20503 rpcz_store.cc:276] 0116 02:59:02.148294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.155076 (+ 6782us) service_pool.cc:225] Handling call
0116 02:59:09.637923 (+7482847us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:02.312107 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:02.312689 (+ 582us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:02.312723 (+ 34us) write_op.cc:435] Acquired schema lock
0116 02:59:02.312742 (+ 19us) tablet.cc:659] Decoding operations
0116 02:59:02.313082 (+ 340us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:02.313166 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:02.313195 (+ 29us) tablet.cc:682] Acquiring locks for 1 operations
0116 02:59:02.313403 (+ 208us) tablet.cc:698] Row locks acquired
0116 02:59:02.313417 (+ 14us) write_op.cc:260] PREPARE: finished
0116 02:59:02.313604 (+ 187us) write_op.cc:270] Start()
0116 02:59:02.313726 (+ 122us) write_op.cc:276] Timestamp: P: 1736996342313537 usec, L: 0
0116 02:59:02.313750 (+ 24us) op_driver.cc:348] REPLICATION: starting
0116 02:59:02.314434 (+ 684us) log.cc:844] Serialized 603 byte log entry
0116 02:59:02.340925 (+ 26491us) op_driver.cc:464] REPLICATION: finished
0116 02:59:02.342027 (+ 1102us) write_op.cc:301] APPLY: starting
0116 02:59:02.342120 (+ 93us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:02.342219 (+ 99us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:02.342351 (+ 132us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:02.379594 (+ 37243us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:02.405188 (+ 25594us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0116 02:59:02.411075 (+ 5887us) write_op.cc:312] APPLY: finished
0116 02:59:02.492020 (+ 80945us) log.cc:844] Serialized 23 byte log entry
0116 02:59:02.520343 (+ 28323us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:02.524130 (+ 3787us) write_op.cc:454] Released schema lock
0116 02:59:02.530953 (+ 6823us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:02.571669 (+ 40716us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:02.575131 (+ 3462us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2050790,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":840,"apply.run_cpu_time_us":3303,"apply.run_wall_time_us":197092,"num_ops":1,"prepare.queue_time_us":79414,"prepare.run_cpu_time_us":2672,"prepare.run_wall_time_us":5388,"replication_time_us":27000,"thread_start_us":75001,"threads_started":1}]]}
I20250116 02:59:09.641214 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:09.642319 20752 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:09.665745 20600 raft_consensus.cc:3049] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:09.920271 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:10.517127 20600 raft_consensus.cc:738] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:10.517836 20607 raft_consensus.cc:2371] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 17446c8dacf04461acb843547665250f in term 2. Re-sending same reply.
I20250116 02:59:10.519949 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.519438 20600 consensus_queue.cc:260] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:10.521044 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1509) took 1583 ms. Trace:
I20250116 02:59:10.521481 20607 rpcz_store.cc:276] 0116 02:59:08.937465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.519517 (+1582052us) service_pool.cc:225] Handling call
0116 02:59:10.520970 (+ 1453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:09.933583 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.524004 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:10.524819 20608 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:09.933938 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.527601 20606 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:10.527733 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:09.680954 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 234) took 6028 ms. Trace:
I20250116 02:59:09.644613 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1717) took 7025 ms. Trace:
I20250116 02:59:09.661581 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:09.516259 21300 connection.cc:232] RPC call timeout handler was delayed by -1.9960912500000632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
I20250116 02:59:09.916559 20572 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:09.639869 21331 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:59:09.662065 21332 raft_consensus.cc:1270] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:10.674108 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 608) took 7281 ms. Trace:
I20250116 02:59:10.674701 21332 rpcz_store.cc:276] 0116 02:59:03.392984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.393185 (+ 201us) service_pool.cc:225] Handling call
0116 02:59:03.438088 (+ 44903us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:09.616886 (+6178798us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54004220a0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326c9f990 00007fc326d11300 00007fc326d12371 00007fc326d0eedf 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc
0116 02:59:10.673401 (+1056515us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.673854 (+ 453us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":321562368}
I20250116 02:59:10.676620 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:09.935174 22241 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:09.637252 22306 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:09.929183 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:09.565129 21328 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
W20250116 02:59:10.284052 20537 debug-util.cc:398] Leaking SignalData structure 0x7b08003d7540 after lost signal to thread 20632
W20250116 02:59:10.305380 21329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 611) took 6821 ms (client timeout 2572 ms). Trace:
W20250116 02:59:10.339756 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 769952us
W20250116 02:59:10.397097 21327 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.007s sys 0.010s
I20250116 02:59:10.402810 20603 raft_consensus.cc:1235] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:10.405277 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:10.413013 20753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1762) took 5618 ms (client timeout 1659 ms). Trace:
I20250116 02:59:10.414096 20754 raft_consensus.cc:1235] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:10.417598 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:10.419938 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:09.532562 22366 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:10.434744 22394 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:10.444164 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.448814 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.451243 20505 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41978 (request call id 324) took 7501 ms. Trace:
I20250116 02:59:10.453315 20491 catalog_manager.cc:4872] TS 17446c8dacf04461acb843547665250f (127.18.238.131:43647): tablet 38583cae882144f48cec12c72ba697cf (table test-workload [id=1fe56c4c593746ebaa1088fc3e6b9437]) successfully deleted
W20250116 02:59:10.482971 21333 consensus_meta.cc:287] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.944s user 0.012s sys 0.016s
W20250116 02:59:10.500396 20751 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.914s user 0.006s sys 0.005s
I20250116 02:59:10.511233 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:10.512965 22325 consensus_meta.cc:287] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.951s user 0.009s sys 0.009s
I20250116 02:59:09.633977 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:10.522295 20600 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:10.529538 20748 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:10.529394 20602 rpcz_store.cc:276] 0116 02:59:02.619757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.625263 (+ 5506us) service_pool.cc:225] Handling call
0116 02:59:02.639205 (+ 13942us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:03.224032 (+584827us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.644312 (+6420280us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b5400253ad4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:09.644543 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1253226880}
I20250116 02:59:10.532644 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.530441 20604 rpcz_store.cc:276] 0116 02:59:03.652748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.663455 (+ 10707us) service_pool.cc:225] Handling call
0116 02:59:09.529004 (+5865549us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:09.587888 (+ 58884us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:09.680725 (+ 92837us) spinlock_profiling.cc:243] Waited 5.86 s on lock 0x7b54002544d0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:09.680890 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13488557952}
I20250116 02:59:10.535391 22356 raft_consensus.cc:2798] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250116 02:59:10.536015 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 618026us
I20250116 02:59:10.542996 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.545006 20608 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
W20250116 02:59:10.549264 22467 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.929s user 0.003s sys 0.010s
I20250116 02:59:10.556525 21300 leader_election.cc:400] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647) with higher term. Message: Invalid argument: T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4.
I20250116 02:59:10.561396 22359 raft_consensus.cc:3049] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
W20250116 02:59:10.567520 20646 connection.cc:232] RPC call timeout handler was delayed by -3.8027836520000164s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25720000000000004s for a response
I20250116 02:59:10.568393 22248 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:10.586678 20647 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:10.590688 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:10.592425 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:10.593140 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:10.594724 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:10.596597 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 242) took 1070 ms. Trace:
I20250116 02:59:10.599077 22468 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:10.600723 21336 raft_consensus.cc:1235] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:10.604699 20720 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:10.611658 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:10.613847 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1410) took 1050 ms. Trace:
W20250116 02:59:10.616727 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 4.788s user 0.021s sys 0.079s
I20250116 02:59:10.628568 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:10.632233 22355 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:10.639808 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1770) took 1920 ms. Trace:
I20250116 02:59:10.641186 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:10.648849 22353 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:10.658689 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:10.744841 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1522) took 1065 ms. Trace:
I20250116 02:59:10.740845 20682 rpcz_store.cc:276] 0116 02:59:09.526173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.662929 (+136756us) service_pool.cc:225] Handling call
0116 02:59:09.663256 (+ 327us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:09.668585 (+ 5329us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.596283 (+927698us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.669014 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:10.678586 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:10.697866 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 246) took 4619 ms. Trace:
I20250116 02:59:10.694301 21299 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:10.720422 20751 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.134s user 0.007s sys 0.005s
W20250116 02:59:10.698860 20753 rpcz_store.cc:269] 0116 02:59:04.794305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:05.761867 (+967562us) service_pool.cc:225] Handling call
0116 02:59:10.412373 (+4650506us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400396554. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:10.412626 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":342630784}
W20250116 02:59:10.694664 21327 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.121s user 0.008s sys 0.010s
W20250116 02:59:10.689970 20537 debug-util.cc:398] Leaking SignalData structure 0x7b08003d5cc0 after lost signal to thread 20707
I20250116 02:59:10.687623 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1514) took 1018 ms. Trace:
I20250116 02:59:10.746445 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1521) took 1065 ms. Trace:
I20250116 02:59:10.741161 21300 leader_election.cc:403] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250116 02:59:10.747785 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 389) took 6915 ms. Trace:
I20250116 02:59:10.745200 20605 rpcz_store.cc:276] 0116 02:59:09.675446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.443463 (+768017us) service_pool.cc:225] Handling call
0116 02:59:10.740978 (+297515us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.742969 20757 rpcz_store.cc:276] 0116 02:59:08.719317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.795298 (+ 75981us) service_pool.cc:225] Handling call
0116 02:59:09.495521 (+700223us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:09.532012 (+ 36491us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.637941 (+1105929us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b540033d1d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:10.639180 (+ 1239us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b540033d1d0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:10.639498 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1691092224}
I20250116 02:59:10.700736 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1389) took 1974 ms. Trace:
W20250116 02:59:10.682924 22328 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20250116 02:59:10.743124 21333 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:10.741679 20677 rpcz_store.cc:276] 0116 02:59:09.563067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.629403 (+ 66336us) service_pool.cc:225] Handling call
0116 02:59:10.613229 (+983826us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b540016e4d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:10.613554 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70634368}
I20250116 02:59:10.753337 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1512) took 1085 ms. Trace:
W20250116 02:59:10.744102 22467 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.124s user 0.006s sys 0.010s
I20250116 02:59:10.748826 20603 rpcz_store.cc:276] 0116 02:59:06.078637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:06.079640 (+ 1003us) service_pool.cc:225] Handling call
0116 02:59:10.401526 (+4321886us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:10.402170 (+ 644us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:10.697454 (+295284us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.697779 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5606272}
I20250116 02:59:10.688283 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56674 (request call id 413) took 1144 ms. Trace:
I20250116 02:59:10.698714 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 251) took 1169 ms. Trace:
I20250116 02:59:10.753849 20607 rpcz_store.cc:276] 0116 02:59:09.668244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.522591 (+854347us) service_pool.cc:225] Handling call
0116 02:59:10.753268 (+230677us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.757984 20678 raft_consensus.cc:2388] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:10.701239 21330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 629) took 1059 ms. Trace:
I20250116 02:59:10.751569 20608 rpcz_store.cc:276] 0116 02:59:09.675188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.523494 (+848306us) service_pool.cc:225] Handling call
0116 02:59:10.740612 (+217118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.752508 21335 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:10.740036 22356 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:10.751766 21336 rpcz_store.cc:276] 0116 02:59:03.832875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.833048 (+ 173us) service_pool.cc:225] Handling call
0116 02:59:10.600050 (+6767002us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:10.600341 (+ 291us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:10.744877 (+144536us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.747684 (+ 2807us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35285760}
W20250116 02:59:10.757018 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 4.929s user 0.022s sys 0.079s
I20250116 02:59:10.708577 20505 rpcz_store.cc:276] 0116 02:59:02.949609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:02.949883 (+ 274us) service_pool.cc:225] Handling call
0116 02:59:10.450992 (+7501109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5934702,"reported_tablets":2}
I20250116 02:59:10.754868 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 250) took 1226 ms. Trace:
I20250116 02:59:10.760267 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 254) took 1202 ms. Trace:
I20250116 02:59:10.743921 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1799) took 1202 ms. Trace:
I20250116 02:59:10.762487 20675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1423) took 1195 ms. Trace:
I20250116 02:59:10.762249 22353 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:10.762887 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 414) took 1202 ms. Trace:
I20250116 02:59:10.763300 22355 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:10.763621 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1806) took 1206 ms. Trace:
I20250116 02:59:10.763864 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:10.764397 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:10.752624 20491 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 299568us
I20250116 02:59:10.764899 22421 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:10.765156 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.765538 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:10.739756 22359 raft_consensus.cc:738] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:10.806489 20749 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:10.791438 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 235257us
I20250116 02:59:10.767238 20680 rpcz_store.cc:276] 0116 02:59:09.557547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.608842 (+ 51295us) service_pool.cc:225] Handling call
0116 02:59:10.640729 (+1031887us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:10.760205 (+119476us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8435456}
W20250116 02:59:10.764335 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 70195.6us
I20250116 02:59:10.767104 20676 rpcz_store.cc:276] 0116 02:59:09.528676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.583220 (+ 54544us) service_pool.cc:225] Handling call
0116 02:59:10.592719 (+1009499us) spinlock_profiling.cc:243] Waited 7.57 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:10.754790 (+162071us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17407360}
I20250116 02:59:10.808279 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:10.808478 20676 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:10.808843 21299 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:10.786800 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 245) took 1259 ms. Trace:
W20250116 02:59:10.775471 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 171390us
I20250116 02:59:10.792997 20601 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:59:10.690168 21329 rpcz_store.cc:269] 0116 02:59:03.484037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.486265 (+ 2228us) service_pool.cc:225] Handling call
0116 02:59:09.513497 (+6027232us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:10.304970 (+791473us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4288896}
I20250116 02:59:10.696638 22341 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20250116 02:59:10.786773 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56674 (request call id 410) took 1250 ms. Trace:
I20250116 02:59:10.809371 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:10.812223 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 256) took 1136 ms. Trace:
I20250116 02:59:10.773423 22356 raft_consensus.cc:3054] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:10.824559 21333 rpcz_store.cc:276] 0116 02:59:09.535979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.536126 (+ 147us) service_pool.cc:225] Handling call
0116 02:59:10.773654 (+1237528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.825420 20681 rpcz_store.cc:276] 0116 02:59:09.620554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.633673 (+ 13119us) service_pool.cc:225] Handling call
0116 02:59:10.753108 (+1119435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.811023 20678 rpcz_store.cc:276] 0116 02:59:09.527189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.620251 (+ 93062us) service_pool.cc:225] Handling call
0116 02:59:10.611041 (+990790us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:10.786705 (+175664us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13347712}
I20250116 02:59:10.828199 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1517) took 1154 ms. Trace:
I20250116 02:59:10.808609 20750 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:10.758675 21328 rpcz_store.cc:276] 0116 02:59:09.543382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.545667 (+ 2285us) service_pool.cc:225] Handling call
0116 02:59:10.688047 (+1142380us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.766208 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 249) took 1237 ms. Trace:
I20250116 02:59:10.742507 22248 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:10.814630 21331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 626) took 1155 ms. Trace:
I20250116 02:59:10.815560 22325 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:10.815800 21299 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:10.816360 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:10.816707 22328 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:10.816715 22366 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:10.819563 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:10.823647 20537 debug-util.cc:398] Leaking SignalData structure 0x7b08003c23a0 after lost signal to thread 20781
I20250116 02:59:10.821040 22359 consensus_queue.cc:260] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:10.822891 22381 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647) with higher term. Message: Invalid argument: T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4.
I20250116 02:59:10.814464 21336 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:10.814464 21335 raft_consensus.cc:2388] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:10.829247 20601 rpcz_store.cc:276] 0116 02:59:09.673992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.448138 (+774146us) service_pool.cc:225] Handling call
0116 02:59:10.828134 (+379996us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.814083 22513 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:10.809458 20676 raft_consensus.cc:604] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:10.824092 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:10.751391 20599 rpcz_store.cc:276] 0116 02:59:09.668710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.927798 (+259088us) service_pool.cc:225] Handling call
0116 02:59:10.687358 (+759560us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.773268 20748 rpcz_store.cc:276] 0116 02:59:09.536561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.526832 (+990271us) service_pool.cc:225] Handling call
0116 02:59:10.723700 (+196868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.772635 21330 rpcz_store.cc:276] 0116 02:59:10.416457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.416866 (+ 409us) service_pool.cc:225] Handling call
0116 02:59:10.701162 (+284296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.795711 22306 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:10.807307 22421 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:10.873842 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.772486 20674 rpcz_store.cc:276] 0116 02:59:09.528974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.583443 (+ 54469us) service_pool.cc:225] Handling call
0116 02:59:10.404637 (+821194us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:10.698574 (+293937us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34714880}
I20250116 02:59:10.781661 20675 rpcz_store.cc:276] 0116 02:59:09.568611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.581701 (+ 13090us) service_pool.cc:225] Handling call
0116 02:59:10.658229 (+1076528us) spinlock_profiling.cc:243] Waited 8.8 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:10.762397 (+104168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20247296}
I20250116 02:59:10.862624 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:10.875052 20749 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:10.875432 20680 raft_consensus.cc:1235] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:10.875605 20753 raft_consensus.cc:1235] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:10.876242 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 243) took 1349 ms. Trace:
I20250116 02:59:10.876407 21300 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:10.876652 20680 rpcz_store.cc:276] 0116 02:59:09.526441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.834042 (+1307601us) service_pool.cc:225] Handling call
0116 02:59:10.875209 (+ 41167us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:10.875924 (+ 715us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.876147 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.877280 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.878435 22346 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:10.879117 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 47599.3us
I20250116 02:59:10.879196 22346 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:10.880066 20720 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:10.881657 20750 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:10.881811 22346 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:10.872566 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:10.868062 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 52371us
I20250116 02:59:10.823992 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.794343 20751 rpcz_store.cc:276] 0116 02:59:09.556776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.584555 (+ 27779us) service_pool.cc:225] Handling call
0116 02:59:09.584919 (+ 364us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:09.585296 (+ 377us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:09.585331 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:09.585840 (+ 509us) log.cc:844] Serialized 26 byte log entry
0116 02:59:10.763177 (+1177337us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:10.763260 (+ 83us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.763340 (+ 80us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:10.763396 (+ 56us) raft_consensus.cc:1680] finished
0116 02:59:10.763407 (+ 11us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:10.763548 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12715,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"thread_start_us":1166552,"threads_started":1,"wal-append.queue_time_us":325306}
W20250116 02:59:10.863675 20537 debug-util.cc:398] Leaking SignalData structure 0x7b08003c23c0 after lost signal to thread 20782
I20250116 02:59:10.753094 20754 rpcz_store.cc:276] 0116 02:59:08.726121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.885502 (+159381us) service_pool.cc:225] Handling call
0116 02:59:10.413459 (+1527957us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:10.699057 (+285598us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.700426 (+ 1369us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b5400396550. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:10.700639 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3512854784}
I20250116 02:59:10.869637 20607 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:10.870434 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.790851 21327 rpcz_store.cc:276] 0116 02:59:09.560261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.571442 (+ 11181us) service_pool.cc:225] Handling call
0116 02:59:09.571893 (+ 451us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:09.572277 (+ 384us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:09.572309 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:09.572864 (+ 555us) log.cc:844] Serialized 26 byte log entry
0116 02:59:10.762358 (+1189494us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:10.762446 (+ 88us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.762545 (+ 99us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:10.762652 (+ 107us) raft_consensus.cc:1680] finished
0116 02:59:10.762662 (+ 10us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:10.762810 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1359,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":117,"thread_start_us":1181703,"threads_started":1,"wal-append.queue_time_us":340289}
I20250116 02:59:10.884557 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.883481 20682 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:10.892489 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1520) took 1214 ms. Trace:
I20250116 02:59:10.892952 20607 rpcz_store.cc:276] 0116 02:59:09.674868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.819152 (+1144284us) service_pool.cc:225] Handling call
0116 02:59:10.889096 (+ 69944us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.893895 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:10.865703 22513 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:10.823472 22468 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:10.869704 22359 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:10.872289 22515 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:10.903867 20606 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:10.886992 22421 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:10.833573 20679 rpcz_store.cc:276] 0116 02:59:09.528398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.582657 (+ 54259us) service_pool.cc:225] Handling call
0116 02:59:10.668426 (+1085769us) spinlock_profiling.cc:243] Waited 8.18 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:10.766138 (+ 97712us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18816384}
I20250116 02:59:10.883190 22325 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:10.914978 20750 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:10.916587 22327 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:10.916951 20572 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:10.917588 22327 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:10.918987 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 178270us
I20250116 02:59:10.920264 22327 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:10.921028 20629 debug-util.cc:398] Leaking SignalData structure 0x7b08001a7640 after lost signal to thread 19389
W20250116 02:59:10.922580 20629 debug-util.cc:398] Leaking SignalData structure 0x7b08000da2a0 after lost signal to thread 20540
I20250116 02:59:10.922080 22515 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:10.924444 20629 debug-util.cc:398] Leaking SignalData structure 0x7b080032f1c0 after lost signal to thread 20632
W20250116 02:59:10.924845 20629 debug-util.cc:398] Leaking SignalData structure 0x7b08000f6ba0 after lost signal to thread 20633
I20250116 02:59:10.903313 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1513) took 1232 ms. Trace:
I20250116 02:59:10.926888 22306 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:10.925570 21332 raft_consensus.cc:2949] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:10.927573 22306 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:10.928184 22484 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:10.927008 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.836434 21331 rpcz_store.cc:276] 0116 02:59:09.638954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.639105 (+ 151us) service_pool.cc:225] Handling call
0116 02:59:09.639505 (+ 400us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:10.715988 (+1076483us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.791107 (+ 75119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.932956 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:10.934870 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 54922us
W20250116 02:59:10.926457 20629 debug-util.cc:398] Leaking SignalData structure 0x7b08002ec540 after lost signal to thread 20707
I20250116 02:59:10.929467 20601 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:10.901588 22513 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:10.944396 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 261) took 1029 ms. Trace:
I20250116 02:59:10.944777 20601 rpcz_store.cc:276] 0116 02:59:09.914992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.878321 (+963329us) service_pool.cc:225] Handling call
0116 02:59:10.944325 (+ 66004us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.925283 20602 rpcz_store.cc:276] 0116 02:59:09.668473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.760915 (+1092442us) service_pool.cc:225] Handling call
0116 02:59:10.900582 (+139667us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.942624 20604 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:10.949384 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1515) took 1280 ms. Trace:
I20250116 02:59:10.949577 20603 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:10.943341 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 252) took 1031 ms. Trace:
I20250116 02:59:10.949841 20604 rpcz_store.cc:276] 0116 02:59:09.668966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.926588 (+1257622us) service_pool.cc:225] Handling call
0116 02:59:10.949292 (+ 22704us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.950378 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 257) took 1036 ms. Trace:
I20250116 02:59:10.950688 20608 rpcz_store.cc:276] 0116 02:59:09.912278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.899259 (+986981us) service_pool.cc:225] Handling call
0116 02:59:10.943258 (+ 43999us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.951296 20603 rpcz_store.cc:276] 0116 02:59:09.913749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.773086 (+859337us) service_pool.cc:225] Handling call
0116 02:59:10.950314 (+177228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.919340 20675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1416) took 1354 ms. Trace:
I20250116 02:59:10.926390 22248 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:10.924926 22515 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:10.971896 22355 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:10.972389 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 55657.1us
I20250116 02:59:10.972709 22355 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:10.973083 22355 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:10.973417 22355 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:10.970198 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 5.25149e+06us
I20250116 02:59:10.958932 20678 raft_consensus.cc:1235] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:10.961905 21299 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:10.951839 22328 raft_consensus.cc:2798] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:10.960397 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.974174 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.983053 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.943261 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.003394 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:11.004889 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 63556.3us
I20250116 02:59:11.007100 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1518) took 1332 ms. Trace:
I20250116 02:59:11.003628 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.003635 20751 raft_consensus.cc:1235] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:11.005790 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1395) took 1430 ms. Trace:
I20250116 02:59:11.012931 20678 rpcz_store.cc:276] 0116 02:59:09.557864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.834802 (+1276938us) service_pool.cc:225] Handling call
0116 02:59:10.909103 (+ 74301us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:10.988018 (+ 78915us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.988317 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1280}
I20250116 02:59:11.009233 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.007496 20608 rpcz_store.cc:276] 0116 02:59:09.674297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.951724 (+1277427us) service_pool.cc:225] Handling call
0116 02:59:11.007037 (+ 55313us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.980504 22328 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:11.018849 20607 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:11.029017 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1510) took 1361 ms. Trace:
I20250116 02:59:11.029449 20607 rpcz_store.cc:276] 0116 02:59:09.667590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.904121 (+1236531us) service_pool.cc:225] Handling call
0116 02:59:11.028934 (+124813us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:10.939267 22366 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.007648 22518 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:11.019826 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1777) took 1374 ms. Trace:
I20250116 02:59:11.020335 20604 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:11.020385 20720 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
W20250116 02:59:11.020161 20537 debug-util.cc:398] Leaking SignalData structure 0x7b08003c2c80 after lost signal to thread 21360
I20250116 02:59:11.036588 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 255) took 1123 ms. Trace:
I20250116 02:59:11.038655 20604 rpcz_store.cc:276] 0116 02:59:09.913062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.956120 (+1043058us) service_pool.cc:225] Handling call
0116 02:59:11.036519 (+ 80399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.039940 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:10.969341 22327 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20250116 02:59:11.037357 20608 rpcz_store.cc:276] 0116 02:59:09.645378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:11.018538 (+1373160us) service_pool.cc:225] Handling call
0116 02:59:11.019173 (+ 635us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:11.019453 (+ 280us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:11.019477 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:11.019522 (+ 45us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:11.019548 (+ 26us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.019605 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:11.019726 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.028147 20601 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:11.044958 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 254) took 1132 ms. Trace:
I20250116 02:59:11.019716 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:11.014830 20629 debug-util.cc:398] Leaking SignalData structure 0x7b08003d25c0 after lost signal to thread 20781
I20250116 02:59:11.056140 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:11.042858 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 277354us
I20250116 02:59:11.036188 22328 raft_consensus.cc:3054] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.043984 20605 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:11.076471 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 251) took 1164 ms. Trace:
I20250116 02:59:11.076849 20605 rpcz_store.cc:276] 0116 02:59:09.911927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.932319 (+1020392us) service_pool.cc:225] Handling call
0116 02:59:11.076393 (+144074us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.053272 20678 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:11.045755 20601 rpcz_store.cc:276] 0116 02:59:09.912799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.950489 (+1037690us) service_pool.cc:225] Handling call
0116 02:59:11.044887 (+ 94398us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.067692 22532 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:11.068073 20681 raft_consensus.cc:1235] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:11.019670 22522 consensus_queue.cc:1046] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
W20250116 02:59:11.055236 20629 debug-util.cc:398] Leaking SignalData structure 0x7b0800311900 after lost signal to thread 20782
I20250116 02:59:11.067773 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.068091 20571 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:11.069964 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 483713us
I20250116 02:59:11.028055 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1400) took 1467 ms. Trace:
W20250116 02:59:11.071398 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 180813us
I20250116 02:59:11.073192 22306 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:10.955957 20675 rpcz_store.cc:276] 0116 02:59:09.564663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.900672 (+1336009us) service_pool.cc:225] Handling call
0116 02:59:10.918520 (+ 17848us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:10.918864 (+ 344us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:10.918909 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:10.918993 (+ 84us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:10.919028 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:10.919106 (+ 78us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:10.919247 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.015906 20599 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:11.090305 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1397) took 1531 ms. Trace:
I20250116 02:59:11.090652 22353 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:11.090056 22532 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:11.111218 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 47851.9us
I20250116 02:59:11.091789 22468 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:11.052328 22347 consensus_queue.cc:1046] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20250116 02:59:11.068295 21299 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:11.106041 20604 raft_consensus.cc:2388] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 2: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:11.108397 20749 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:11.111495 20681 rpcz_store.cc:276] 0116 02:59:09.558449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.839325 (+1280876us) service_pool.cc:225] Handling call
0116 02:59:11.051704 (+212379us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98d0e 0000000000353db9 00007fc32bbf8a70 00007fc32bcbe834 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:11.051963 (+ 259us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:11.089944 (+ 37981us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.090223 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5572864}
I20250116 02:59:11.026435 22484 raft_consensus.cc:695] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:11.117569 20680 raft_consensus.cc:1212] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
I20250116 02:59:11.127902 20680 raft_consensus.cc:1235] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20250116 02:59:11.054857 22341 consensus_queue.cc:1046] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20250116 02:59:11.121874 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.123805 20753 raft_consensus.cc:1235] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:11.139096 20720 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:11.140693 22356 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.141451 20608 raft_consensus.cc:1235] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:11.141582 22484 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.143151 20683 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:11.145872 22366 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:11.090090 22526 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:11.104552 20674 rpcz_store.cc:276] 0116 02:59:09.560295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.891167 (+1330872us) service_pool.cc:225] Handling call
0116 02:59:10.952333 (+ 61166us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:10.974228 (+ 21895us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:10.980180 (+ 5952us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:10.987329 (+ 7149us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:11.006016 (+ 18687us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.009806 (+ 3790us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:11.017806 (+ 8000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.151648 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1779) took 1503 ms. Trace:
I20250116 02:59:11.152240 20607 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:11.053272 22421 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.130528 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1398) took 1571 ms. Trace:
I20250116 02:59:11.120622 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 259) took 1204 ms. Trace:
W20250116 02:59:11.157229 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 183558us
I20250116 02:59:11.159459 20608 rpcz_store.cc:276] 0116 02:59:09.645850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:11.135530 (+1489680us) service_pool.cc:225] Handling call
0116 02:59:11.141290 (+ 5760us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:11.149401 (+ 8111us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.149618 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.163753 22413 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:11.166138 20680 rpcz_store.cc:276] 0116 02:59:09.558794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.877525 (+1318731us) service_pool.cc:225] Handling call
0116 02:59:10.878196 (+ 671us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:11.128358 (+250162us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.130176 (+ 1818us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54001c4dd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:11.130449 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":550128000}
I20250116 02:59:11.169824 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1519) took 1495 ms. Trace:
I20250116 02:59:11.170939 20607 rpcz_store.cc:276] 0116 02:59:09.674557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:11.106935 (+1432378us) service_pool.cc:225] Handling call
0116 02:59:11.147669 (+ 40734us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:11.169521 (+ 21852us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.169745 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.171881 20605 raft_consensus.cc:1235] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:11.172407 20607 raft_consensus.cc:3049] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:11.172614 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 262) took 1257 ms. Trace:
I20250116 02:59:11.173158 20607 raft_consensus.cc:738] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:11.173143 20605 rpcz_store.cc:276] 0116 02:59:09.915321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:11.165901 (+1250580us) service_pool.cc:225] Handling call
0116 02:59:11.171683 (+ 5782us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:11.172303 (+ 620us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.172525 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:11.165525 20629 debug-util.cc:398] Leaking SignalData structure 0x7b0800203ae0 after lost signal to thread 21360
I20250116 02:59:11.174693 20607 consensus_queue.cc:260] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:11.165621 20647 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.608s (SENT) [suppressed 31 similar messages]
W20250116 02:59:11.191704 20683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 217) took 7880 ms (client timeout 1907 ms). Trace:
I20250116 02:59:11.164831 22526 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:11.192188 20683 rpcz_store.cc:269] 0116 02:59:03.305318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:03.310989 (+ 5671us) service_pool.cc:225] Handling call
0116 02:59:11.170278 (+7859289us) spinlock_profiling.cc:243] Waited 7.79 s on lock 0x7b5400087ad4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:11.185344 (+ 15066us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17919079296}
I20250116 02:59:11.166882 22325 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:11.200953 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 73271.7us
W20250116 02:59:11.205857 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 137686us
I20250116 02:59:11.212937 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 249) took 1297 ms. Trace:
I20250116 02:59:11.166746 20599 rpcz_store.cc:276] 0116 02:59:09.914382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.872930 (+958548us) service_pool.cc:225] Handling call
0116 02:59:11.118821 (+245891us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.213603 20604 rpcz_store.cc:276] 0116 02:59:09.911273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:11.137567 (+1226294us) service_pool.cc:225] Handling call
0116 02:59:11.185783 (+ 48216us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:11.202954 (+ 17171us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:11.202979 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:11.207870 (+ 4891us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:11.207915 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.207989 (+ 74us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:11.208144 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:11.227887 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 89052.9us
I20250116 02:59:11.210645 20602 raft_consensus.cc:2388] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:11.201403 22526 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:11.192644 22366 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:11.232015 21299 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:11.233033 22391 raft_consensus.cc:3049] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:11.233624 22353 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:11.234916 20607 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.238960 22538 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:11.209013 20600 raft_consensus.cc:1270] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:59:11.242970 22441 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:11.272418 22441 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.244112 22532 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:11.273556 21299 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:11.244392 22325 consensus_queue.cc:1046] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:11.234244 22347 raft_consensus.cc:3049] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
W20250116 02:59:11.249935 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 107605us
W20250116 02:59:11.251917 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 184016us
I20250116 02:59:11.252080 22544 consensus_queue.cc:1046] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:11.253916 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 263) took 1338 ms. Trace:
W20250116 02:59:11.244428 22537 log.cc:926] Time spent T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250116 02:59:11.265556 20720 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:11.278864 22538 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:11.269125 20629 debug-util.cc:398] Leaking SignalData structure 0x7b08003ca100 after lost signal to thread 22475
I20250116 02:59:11.260170 22391 raft_consensus.cc:738] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:11.268492 22467 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:11.286895 22518 raft_consensus.cc:2743] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:11.288249 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250116 02:59:11.290983 22325 raft_consensus.cc:3049] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:11.280483 20602 rpcz_store.cc:276] 0116 02:59:09.915618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.959194 (+1043576us) service_pool.cc:225] Handling call
0116 02:59:11.253634 (+294440us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54000573d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:11.253835 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":295051136}
I20250116 02:59:11.292111 22353 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.293583 22413 consensus_queue.cc:1046] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:11.294840 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1776) took 1633 ms. Trace:
I20250116 02:59:11.295157 22346 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:11.277612 22347 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:11.299224 22534 consensus_queue.cc:1046] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20250116 02:59:11.266746 22468 raft_consensus.cc:3049] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:11.280603 22341 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:11.301621 22356 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:11.303074 20571 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:11.302762 22355 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.307103 22421 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:11.313544 22532 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250116 02:59:11.313849 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 53947.3us
I20250116 02:59:11.334668 22467 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.336117 22538 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:11.331153 20629 debug-util.cc:398] Leaking SignalData structure 0x7b0800265ae0 after lost signal to thread 22478
I20250116 02:59:11.337566 22529 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:11.339088 22530 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:11.340456 20606 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:11.338958 20677 raft_consensus.cc:1235] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:11.373045 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250116 02:59:11.383332 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 80450.8us
W20250116 02:59:11.387758 20606 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 265) took 1471 ms (client timeout 1762 ms). Trace:
I20250116 02:59:11.342936 21336 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
W20250116 02:59:11.389531 20606 rpcz_store.cc:269] 0116 02:59:09.916187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.766320 (+850133us) service_pool.cc:225] Handling call
0116 02:59:11.387652 (+621332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.360859 22413 consensus_queue.cc:1225] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer 231d91433a404468a44ea0a107345aeb log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250116 02:59:11.333865 22325 raft_consensus.cc:738] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:11.366022 22341 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:11.388149 20629 debug-util.cc:398] Leaking SignalData structure 0x7b08003a6200 after lost signal to thread 22479
I20250116 02:59:11.363202 22468 raft_consensus.cc:738] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
W20250116 02:59:11.398034 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 116769us
W20250116 02:59:11.343690 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 97356.7us
W20250116 02:59:11.379799 22394 consensus_meta.cc:287] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.656s user 0.017s sys 0.000s
I20250116 02:59:11.392918 21336 pending_rounds.cc:77] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Aborting all ops after (but not including) 3
I20250116 02:59:11.404650 22356 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:11.374441 22328 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.346140 20600 rpcz_store.cc:276] 0116 02:59:09.645136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:09.664874 (+ 19738us) service_pool.cc:225] Handling call
0116 02:59:09.665312 (+ 438us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:11.274249 (+1608937us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.278291 (+ 4042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:11.410236 22240 raft_consensus.cc:3049] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:11.410727 22240 raft_consensus.cc:738] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:11.366991 22532 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:11.412904 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1396) took 1854 ms. Trace:
I20250116 02:59:11.405869 21336 pending_rounds.cc:99] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20250116 02:59:11.369679 22353 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:11.417707 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 135783us
I20250116 02:59:11.416054 20677 rpcz_store.cc:276] 0116 02:59:09.558145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.836758 (+1278613us) service_pool.cc:225] Handling call
0116 02:59:10.893173 (+ 56415us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:11.388310 (+495137us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.406074 (+ 17764us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b5400323a54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:11.412835 (+ 6761us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1024469376}
I20250116 02:59:11.416124 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250116 02:59:11.417750 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 144269us
I20250116 02:59:11.396765 22546 consensus_queue.cc:1046] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20250116 02:59:11.416188 21336 raft_consensus.cc:2911] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20250116 02:59:11.405019 22341 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:11.418334 20720 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:11.424654 21336 raft_consensus.cc:2883] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:11.421710 20756 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:11.417898 22532 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:11.427524 21302 leader_election.cc:336] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.762s (SENT)
I20250116 02:59:11.374432 22526 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.376081 22347 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.443693 20571 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:11.444571 20720 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:11.447263 20723 leader_election.cc:336] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.609s (SENT)
I20250116 02:59:11.453112 21299 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:11.451864 20723 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:11.459719 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1778) took 1809 ms. Trace:
W20250116 02:59:11.447762 22545 log.cc:926] Time spent T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20250116 02:59:11.368193 22542 consensus_queue.cc:1046] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.181s
I20250116 02:59:11.452657 21302 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:11.425153 22413 consensus_queue.cc:1046] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250116 02:59:11.446795 22346 raft_consensus.cc:3049] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:11.461540 20601 rpcz_store.cc:276] 0116 02:59:09.645608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:11.126068 (+1480460us) service_pool.cc:225] Handling call
0116 02:59:11.138254 (+ 12186us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:11.206834 (+ 68580us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:11.206866 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:11.232974 (+ 26108us) log.cc:844] Serialized 26 byte log entry
0116 02:59:11.343126 (+110152us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:11.343167 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:11.343237 (+ 70us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:11.448893 (+105656us) raft_consensus.cc:1680] finished
0116 02:59:11.448914 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:11.454908 (+ 5994us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":67819,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":19333,"thread_start_us":170995,"threads_started":2,"wal-append.queue_time_us":184363}
I20250116 02:59:11.443540 22328 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:11.464223 22353 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250116 02:59:11.481245 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 61639.1us
I20250116 02:59:11.490219 20571 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:11.491057 22542 consensus_queue.cc:1225] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer 17446c8dacf04461acb843547665250f log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250116 02:59:11.467763 22394 raft_consensus.cc:2758] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250116 02:59:11.396983 22530 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.448750 22534 consensus_queue.cc:1035] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250116 02:59:11.429668 22240 consensus_queue.cc:260] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.448014 22513 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.479625 22346 raft_consensus.cc:738] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:11.522744 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 78254.5us
I20250116 02:59:11.447319 22468 consensus_queue.cc:260] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.496009 22441 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:11.547814 22441 consensus_queue.cc:1035] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:11.525018 22544 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.538154 22540 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:11.538496 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:11.539242 22437 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:11.539840 22561 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250116 02:59:11.540019 22359 consensus_meta.cc:287] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.635s user 0.015s sys 0.001s
I20250116 02:59:11.571346 22437 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.537490 22355 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:11.573904 22530 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:11.574270 22240 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:11.575229 22346 consensus_queue.cc:260] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.573962 20720 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:11.577359 21335 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:11.541749 20646 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:11.566715 22526 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:11.568763 22513 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:11.569698 22356 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:11.589588 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 153454us
I20250116 02:59:11.540603 22551 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:11.587076 20753 raft_consensus.cc:3054] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.538154 22334 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEARNER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:11.547292 22467 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:11.562028 22381 raft_consensus.cc:2798] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250116 02:59:11.590420 20723 leader_election.cc:336] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.759s (SENT)
I20250116 02:59:11.500108 22325 consensus_queue.cc:260] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.603096 20723 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:11.604154 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 62524.4us
I20250116 02:59:11.606849 22549 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:11.601811 22530 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:11.611999 20756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1785) took 2672 ms (client timeout 1608 ms). Trace:
W20250116 02:59:11.612601 20756 rpcz_store.cc:269] 0116 02:59:08.939205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:08.974183 (+ 34978us) service_pool.cc:225] Handling call
0116 02:59:11.611690 (+2637507us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b54003723d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:11.611882 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5616064256}
I20250116 02:59:11.590669 22355 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:11.618907 22325 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:11.536919 22391 consensus_queue.cc:260] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.631057 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.631512 22437 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:11.632102 21299 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:11.632593 22347 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.632699 22513 raft_consensus.cc:3054] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:11.640125 22341 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.595204 22346 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.645846 20646 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:11.647445 22417 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:11.647958 22417 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:11.640476 20720 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:11.659993 22391 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:11.661175 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 96081.1us
I20250116 02:59:11.665037 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.666049 20599 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:11.666181 20720 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:11.668047 22353 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:11.668458 20720 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:11.669404 22437 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:11.670163 22437 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:11.670341 20720 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
W20250116 02:59:11.675573 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 399287us
I20250116 02:59:11.676153 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.677078 20607 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:11.642509 22341 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:11.653200 22417 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:11.682757 22325 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:11.674490 22437 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:11.690447 22325 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.693085 22325 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:11.683087 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:11.694764 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.690989 22437 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.701048 22468 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:11.685498 20649 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:11.703595 22534 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:11.705761 22534 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:11.706486 22534 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:11.705538 20649 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:11.711043 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 221015us
I20250116 02:59:11.707933 20753 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:11.712746 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.713348 22543 raft_consensus.cc:3049] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:11.713752 22543 raft_consensus.cc:738] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:11.714962 22355 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:11.714749 22543 consensus_queue.cc:260] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:11.715404 22355 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:11.715780 22355 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.716225 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:11.716907 20749 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:11.719198 22543 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:59:11.721693 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 64582.7us
I20250116 02:59:11.723891 22327 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:59:11.724954 22516 raft_consensus.cc:2798] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:11.695714 20600 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:11.721756 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.714915 22341 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:11.725452 22516 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:11.727130 22341 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.727862 22516 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.729598 22341 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:11.730475 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 98477.4us
I20250116 02:59:11.727154 22306 raft_consensus.cc:988] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:59:11.731552 22441 raft_consensus.cc:1076] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
W20250116 02:59:11.721925 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 261282us
I20250116 02:59:11.725212 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.735208 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 260) took 1820 ms. Trace:
I20250116 02:59:11.748507 20603 rpcz_store.cc:276] 0116 02:59:09.914685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:10.958157 (+1043472us) service_pool.cc:225] Handling call
0116 02:59:11.734986 (+776829us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b54001f09d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:11.735136 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1327337344}
I20250116 02:59:11.754662 22441 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:11.756531 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:11.756758 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:11.757349 20756 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:11.757460 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:11.756376 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:11.758430 21332 raft_consensus.cc:2371] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3. Re-sending same reply.
I20250116 02:59:11.755012 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:11.760532 21300 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:11.779191 22391 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:11.782953 22391 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:11.783422 22391 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:11.785221 22529 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:11.786823 21332 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:11.787264 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:11.788009 21335 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:11.789546 22544 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:11.790447 20604 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:11.791054 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:11.793960 20677 raft_consensus.cc:1235] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:11.797312 20602 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:11.790953 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:11.799120 20605 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.801685 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:11.802639 21335 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:11.805135 22538 raft_consensus.cc:3049] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:11.833213 22538 raft_consensus.cc:738] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:11.839489 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:11.853492 20646 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:11.725487 22327 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.115s
I20250116 02:59:11.880461 22518 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:11.881129 22518 raft_consensus.cc:695] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:11.884750 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 104679us
I20250116 02:59:11.861364 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:11.886694 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.888008 21300 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:11.880709 22381 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:11.905812 22562 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:11.906432 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 70509.7us
I20250116 02:59:11.885695 22421 raft_consensus.cc:3049] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:11.809221 20505 catalog_manager.cc:5526] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:11.901909 20749 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:11.916527 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 70826.5us
I20250116 02:59:11.897981 22334 raft_consensus.cc:2798] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:11.924401 22334 raft_consensus.cc:695] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:11.925025 22513 raft_consensus.cc:2743] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:11.925585 22334 consensus_queue.cc:237] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.914170 22421 raft_consensus.cc:738] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:11.926982 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:11.927578 22421 consensus_queue.cc:260] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.928473 21330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56664
I20250116 02:59:11.929327 21330 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:59:11.929991 21330 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:11.931602 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.940605 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.941432 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.941977 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.942332 20606 raft_consensus.cc:2388] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:11.939688 22353 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:11.943379 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:11.947012 20606 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.947449 22437 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.950166 22437 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:11.834537 22538 consensus_queue.cc:260] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:11.911854 22381 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:11.955394 20505 catalog_manager.cc:5526] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 3 to 4, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 4 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:11.956746 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.981568 20604 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:11.959146 20601 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:11.967482 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:11.968257 21300 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:11.966769 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:11.969503 20646 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:11.971509 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.972499 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.999320 21333 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:11.962319 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.006474 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.006485 20606 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
W20250116 02:59:12.010285 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 167085us
I20250116 02:59:12.010437 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.011693 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.012375 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.013324 20606 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:12.012545 20505 catalog_manager.cc:5526] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:59:12.021349 20505 catalog_manager.cc:5526] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:59:12.029947 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.031503 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.032239 20601 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:12.032935 21302 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:12.034780 22328 consensus_queue.cc:1046] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:12.035353 22562 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:12.035897 22562 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:12.036316 22562 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.039551 21302 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.041229 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.036159 22328 raft_consensus.cc:3049] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:11.960758 22306 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:12.041493 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:12.041929 22552 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:11.972273 22518 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.041962 20608 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:12.046774 20606 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:12.042304 22328 raft_consensus.cc:738] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:12.048739 22328 consensus_queue.cc:260] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.049407 22391 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.050453 22328 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:11.962154 21331 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 3. Ops: []
I20250116 02:59:11.954533 22516 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:12.052584 22516 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:12.006578 22566 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:12.054177 20606 raft_consensus.cc:1235] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:11.978691 22355 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:12.042411 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:11.975764 22532 consensus_meta.cc:287] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.513s user 0.001s sys 0.028s
I20250116 02:59:12.027117 20681 raft_consensus.cc:1235] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:11.974218 22538 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:12.056983 22347 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:12.061275 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:11.973241 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.056896 22355 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:12.068442 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.088690 20607 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:12.027462 22441 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:12.082394 21302 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:12.064222 20505 catalog_manager.cc:5526] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:11.970185 22327 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20250116 02:59:12.097615 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.096845 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.070379 22347 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:12.098518 20753 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:12.093832 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.103919 20606 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.062922 22566 raft_consensus.cc:695] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:12.106438 20603 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:12.042814 22306 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.111274 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.109642 20605 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:12.099350 20505 catalog_manager.cc:5526] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:12.034749 22346 consensus_queue.cc:1046] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20250116 02:59:12.115819 21302 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:12.100862 20723 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.118090 20681 raft_consensus.cc:1235] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:12.093883 22441 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:12.126394 20505 catalog_manager.cc:5526] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:12.131693 20505 catalog_manager.cc:5526] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:12.106168 22347 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:12.131749 20756 raft_consensus.cc:1235] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:12.134375 20505 catalog_manager.cc:5526] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 3 to 4, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 4 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:12.107422 22413 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:12.145040 20606 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:12.144608 20505 catalog_manager.cc:5526] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:12.072925 22532 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.148555 20681 raft_consensus.cc:1235] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:12.149223 20683 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
W20250116 02:59:12.150067 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 67001us
I20250116 02:59:12.150889 20723 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.069999 22534 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.131909 22441 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.147221 20505 catalog_manager.cc:5526] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 2 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:12.140864 22355 raft_consensus.cc:2743] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:12.149906 22391 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:12.159282 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.157635 20505 catalog_manager.cc:5526] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:59:12.143610 22413 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:12.140313 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.163015 20752 raft_consensus.cc:1235] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:12.129084 22417 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.171938 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:12.175705 20756 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:12.177908 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.180749 20505 catalog_manager.cc:5526] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:12.175832 20723 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.184916 20505 catalog_manager.cc:5526] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:12.187453 20680 raft_consensus.cc:2363] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4.
I20250116 02:59:12.190124 20505 catalog_manager.cc:5526] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:12.158802 22437 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:12.200348 20678 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:12.220525 20683 raft_consensus.cc:1235] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:12.176645 22413 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.171147 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.316970 22381 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.317644 22421 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.321966 22565 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:12.322463 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 206752us
I20250116 02:59:12.241904 22566 consensus_queue.cc:237] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.331369 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:12.336092 22575 log.cc:926] Time spent T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20250116 02:59:12.330972 22563 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:12.242672 20600 raft_consensus.cc:2388] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:12.346046 21299 leader_election.cc:400] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4.
I20250116 02:59:12.403234 21299 leader_election.cc:403] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250116 02:59:12.347277 20756 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:12.354003 20601 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:12.403709 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.404044 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.359766 22355 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:12.381350 22577 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.187s user 0.000s sys 0.008s
I20250116 02:59:12.364768 20680 raft_consensus.cc:2388] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:12.366129 22381 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:12.366931 21335 raft_consensus.cc:1235] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250116 02:59:12.407205 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.410518 21302 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.413025 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.413977 20755 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:12.414038 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.367543 20674 raft_consensus.cc:1235] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:12.414777 20756 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
W20250116 02:59:12.368438 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 193132us
I20250116 02:59:12.415771 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.370040 22565 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.416512 20751 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
W20250116 02:59:12.344072 22578 log.cc:926] Time spent T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.099s user 0.000s sys 0.005s
I20250116 02:59:12.372258 22417 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:12.372114 22347 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.367102 22306 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:12.373839 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 198077us
W20250116 02:59:12.378906 22576 log.cc:926] Time spent T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.217s user 0.000s sys 0.010s
I20250116 02:59:12.379555 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:12.368353 20677 raft_consensus.cc:1235] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:12.383258 20602 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:59:12.383463 22563 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:12.393723 22515 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:12.396068 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.397434 20649 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:12.397429 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.398943 22341 raft_consensus.cc:3049] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:12.401486 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.407138 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:12.418375 22565 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:12.413112 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.444933 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.444453 22341 raft_consensus.cc:738] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:12.445082 21333 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:12.429563 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.428663 22381 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:12.431520 20755 raft_consensus.cc:1270] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250116 02:59:12.447983 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 50651.5us
I20250116 02:59:12.381572 21330 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.448701 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.449318 22516 raft_consensus.cc:2798] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:12.449990 22516 raft_consensus.cc:695] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:12.450938 22516 consensus_queue.cc:237] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:12.452167 20756 raft_consensus.cc:1270] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:59:12.452688 20723 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.436127 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.432840 20757 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:12.446296 22341 consensus_queue.cc:260] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.410638 21334 raft_consensus.cc:1235] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 4. Ops: []
I20250116 02:59:12.455262 22515 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:12.456871 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.431977 20750 raft_consensus.cc:1270] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:59:12.447317 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.458065 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:12.458271 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 47840.1us
I20250116 02:59:12.458801 20751 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:12.454267 22355 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:12.459223 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:12.428314 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 82421.6us
I20250116 02:59:12.460629 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.463512 21333 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:12.465909 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.454046 20723 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.462725 20676 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:12.424374 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.467582 20723 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:12.447790 22579 log.cc:926] Time spent T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20250116 02:59:12.469399 20723 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:12.461632 22341 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.446831 20683 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:12.464028 20753 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:12.468475 22437 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:12.469419 22355 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:12.473258 21333 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:12.474828 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.475524 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.433926 21300 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:12.457571 21334 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:12.453981 22563 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.496217 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.484393 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.497347 20753 raft_consensus.cc:2363] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:12.436702 22346 consensus_queue.cc:1046] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20250116 02:59:12.486853 21330 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:12.487270 20677 raft_consensus.cc:1235] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:12.446358 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.487735 22334 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:12.380089 22562 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.489100 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.489845 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.490530 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:12.490954 22441 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.491350 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.492003 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.453975 20748 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:12.487978 20571 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:12.449373 20755 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:12.503196 21302 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.457518 22515 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:12.504712 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:12.496251 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 62434.9us
I20250116 02:59:12.500649 22346 raft_consensus.cc:3049] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:12.512207 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.512392 20677 raft_consensus.cc:1235] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
W20250116 02:59:12.514601 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 408974us
I20250116 02:59:12.434479 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.498700 22306 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:12.532979 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.554379 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.511160 22437 raft_consensus.cc:2798] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:12.566982 22437 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:12.567519 22437 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.570257 22441 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:12.502038 22565 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
W20250116 02:59:12.552196 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 90581.4us
I20250116 02:59:12.555112 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.555598 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.554823 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.526170 21328 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:12.556056 21331 raft_consensus.cc:1235] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:12.532893 22346 raft_consensus.cc:738] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:12.516119 22347 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:12.552548 20751 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:12.580920 21332 raft_consensus.cc:2388] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:12.588089 22241 consensus_queue.cc:1046] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20250116 02:59:12.589300 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.591236 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:12.591919 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.593276 21336 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:12.596702 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 93582.2us
I20250116 02:59:12.546308 22518 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:12.550077 22543 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:12.601454 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.583887 22421 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:12.608448 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.612975 20605 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:12.590148 22534 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:12.603904 22587 log.cc:926] Time spent T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20250116 02:59:12.589416 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:12.619792 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:12.624637 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:12.626842 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 138970us
I20250116 02:59:12.631525 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.606050 22532 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:12.635207 20571 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:12.586730 22306 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:12.604835 22565 raft_consensus.cc:695] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:12.630769 20605 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
W20250116 02:59:12.638990 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 169654us
I20250116 02:59:12.612119 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:12.639678 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 105429us
I20250116 02:59:12.637110 22248 raft_consensus.cc:3049] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:12.666275 22248 raft_consensus.cc:738] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:12.667429 22248 consensus_queue.cc:260] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.671703 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.672916 20679 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:12.673662 20723 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.675310 22591 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.675922 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:12.676362 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 48530.3us
I20250116 02:59:12.681043 20646 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:12.681805 20681 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:12.687533 20723 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:12.693186 20723 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.695340 22534 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:12.612526 22518 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:12.710809 21332 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:12.658764 22543 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:12.725266 20723 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:12.618968 22346 consensus_queue.cc:260] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.708527 20674 raft_consensus.cc:3049] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:12.704278 22532 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:12.709527 22534 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:12.747747 20749 raft_consensus.cc:1235] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:12.650395 22413 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.752135 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.752314 20605 raft_consensus.cc:1235] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:12.763968 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.764678 20605 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.753651 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.738766 22546 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:12.755612 21302 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.717520 22355 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:12.697683 22538 consensus_queue.cc:1046] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20250116 02:59:12.761759 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.768756 22534 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:12.769515 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.755506 20755 raft_consensus.cc:1270] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250116 02:59:12.770682 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.782574 20606 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:12.773047 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.783913 20601 raft_consensus.cc:2371] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 17446c8dacf04461acb843547665250f in term 3. Re-sending same reply.
I20250116 02:59:12.770654 21333 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:12.778753 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:12.749641 22562 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:12.779822 22355 raft_consensus.cc:2798] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:12.785380 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:12.749559 20674 raft_consensus.cc:738] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:12.788045 20683 raft_consensus.cc:1270] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250116 02:59:12.790477 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 112283us
W20250116 02:59:12.791170 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 108643us
I20250116 02:59:12.802050 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:12.802850 21333 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:12.784493 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 147430us
I20250116 02:59:12.788406 20605 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:12.802366 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:12.783509 22341 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:12.822516 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.824793 20755 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:12.782737 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.818397 21333 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:12.845685 21334 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:12.758316 22565 consensus_queue.cc:237] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.795495 22355 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:12.799669 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.850502 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.823279 22306 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:12.875872 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.850906 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.886910 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.797324 22546 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.850436 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.849972 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.848997 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:12.878455 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:12.878299 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 51811us
I20250116 02:59:12.884334 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:12.883692 21302 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:12.881384 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 103243us
I20250116 02:59:12.887578 20607 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:12.869119 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:12.894445 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.870831 22381 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4.
I20250116 02:59:12.866652 20602 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:12.896301 21302 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.916293 20755 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:12.876271 22355 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:12.934274 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 209190us
I20250116 02:59:12.945904 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:12.881489 22600 raft_consensus.cc:3049] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Stepping down as leader of term 2
W20250116 02:59:12.935861 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 55352.7us
I20250116 02:59:12.935210 21302 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.951974 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:12.943097 20603 raft_consensus.cc:1270] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250116 02:59:12.868468 20674 consensus_queue.cc:260] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:12.827478 22543 consensus_queue.cc:1046] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20250116 02:59:12.952752 20751 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:12.952695 20607 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:12.954656 20604 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:12.964088 22532 raft_consensus.cc:695] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:12.874387 22538 consensus_queue.cc:1046] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20250116 02:59:12.953913 20749 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:12.897588 22594 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:12.897459 22341 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:12.950567 22240 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:12.840014 22597 consensus_queue.cc:1046] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.138s
I20250116 02:59:12.974579 20571 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:12.945632 22552 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:12.982476 20723 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:12.983592 20720 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:12.985984 20647 leader_election.cc:336] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.608s (SENT)
I20250116 02:59:12.979367 22421 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20250116 02:59:12.987897 20674 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:12.991156 21329 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:12.970585 21302 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:12.950802 22600 raft_consensus.cc:738] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:12.994989 20646 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:12.994421 21332 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:12.994225 20751 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:13.004060 20683 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:12.975890 22381 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:12.992484 22601 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:13.073230 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 102727us
I20250116 02:59:13.001104 22518 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:13.083631 21336 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:13.075546 21302 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.000602 22341 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:13.001130 22240 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.124892 22600 consensus_queue.cc:260] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.010020 22413 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:13.019764 22596 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:13.143954 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:59:13.023217 22594 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250116 02:59:13.144918 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 161370us
W20250116 02:59:13.137717 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 162116us
I20250116 02:59:13.157157 20649 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:13.157835 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 82365.4us
I20250116 02:59:13.158927 21302 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.166584 20604 raft_consensus.cc:1235] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:13.174700 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:13.175736 20607 raft_consensus.cc:2388] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:12.974224 22532 consensus_queue.cc:237] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.177057 20720 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.189046 21302 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:13.175294 22328 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:13.191264 22328 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:13.191145 22562 consensus_queue.cc:1035] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:13.191813 22328 raft_consensus.cc:695] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:13.192832 22328 consensus_queue.cc:237] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.127343 22341 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:13.195109 22588 log.cc:926] Time spent T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.407s user 0.001s sys 0.001s
I20250116 02:59:13.196641 22341 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:13.197021 22596 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.201328 22240 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:13.202836 22240 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:13.203394 22240 raft_consensus.cc:695] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:13.204348 22610 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:13.187155 22381 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250116 02:59:13.207850 22437 consensus_meta.cc:287] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.640s user 0.012s sys 0.000s
I20250116 02:59:13.208369 22437 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.209950 20753 raft_consensus.cc:2388] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:13.182998 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:13.216812 21302 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.190277 20604 raft_consensus.cc:3049] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:13.185743 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:13.207508 22609 consensus_queue.cc:1046] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:13.217926 20604 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:13.218999 20607 raft_consensus.cc:2388] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:13.165390 22599 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
W20250116 02:59:13.225509 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 48602.5us
I20250116 02:59:13.182267 22543 raft_consensus.cc:3049] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:13.255353 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.205034 22610 raft_consensus.cc:695] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:13.211244 22604 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election won for term 3
W20250116 02:59:13.257546 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 247192us
I20250116 02:59:13.250720 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:13.210551 22437 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:13.239348 22381 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:13.278100 20674 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
W20250116 02:59:13.288015 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 305583us
I20250116 02:59:13.109014 22601 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.263603 20604 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.292220 20604 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:13.250123 22543 raft_consensus.cc:738] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:13.297446 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 343757us
I20250116 02:59:13.300169 20723 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:13.300791 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.204500 22240 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.267465 22599 raft_consensus.cc:695] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:13.279939 22546 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:13.274492 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:13.301663 20753 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:13.314882 21300 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:13.311355 22597 raft_consensus.cc:2743] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:13.294209 22347 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20250116 02:59:13.316138 22617 raft_consensus.cc:3049] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Stepping down as leader of term 3
W20250116 02:59:13.343726 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 97929.5us
I20250116 02:59:13.352453 20571 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:13.317181 22619 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:13.318195 22596 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:13.317597 22612 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:13.328986 22546 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250116 02:59:13.357769 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 86498.7us
W20250116 02:59:13.372043 22590 log.cc:926] Time spent T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.458s user 0.007s sys 0.013s
I20250116 02:59:13.351498 22610 consensus_queue.cc:237] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.373833 22607 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.162s
I20250116 02:59:13.326145 22355 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.374159 22346 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:13.352360 22617 raft_consensus.cc:738] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:13.363611 22546 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:13.436023 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 45884.5us
W20250116 02:59:13.444854 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 136665us
I20250116 02:59:13.380314 22532 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:13.378322 22619 raft_consensus.cc:695] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:13.447620 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:13.450331 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:13.451540 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 46890us
I20250116 02:59:13.453094 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:13.382367 22597 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:13.457595 20751 raft_consensus.cc:1235] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
W20250116 02:59:13.460193 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 113418us
I20250116 02:59:13.462512 20683 raft_consensus.cc:2388] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 2: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:13.462915 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:13.463618 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:13.464308 20683 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:13.464601 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:13.465601 22618 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:13.465744 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:13.466251 20680 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:13.466202 22618 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.464218 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:13.470561 20678 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:13.480154 20608 raft_consensus.cc:1270] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:13.484640 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:13.485489 20676 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:13.449589 22546 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:13.488804 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:13.480376 20723 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.391150 22543 consensus_queue.cc:260] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.499167 22623 consensus_queue.cc:1046] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20250116 02:59:13.500774 22618 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:13.374410 22612 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.501793 21334 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:13.469066 22532 raft_consensus.cc:695] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:13.502907 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:13.503227 22532 consensus_queue.cc:237] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.504221 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:13.501073 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:13.505717 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.501288 22619 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.500533 22543 raft_consensus.cc:3054] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:13.494359 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.501730 20571 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:13.507908 20753 raft_consensus.cc:1270] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:59:13.524296 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:13.514474 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.523984 20755 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:13.525131 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 44592.8us
I20250116 02:59:13.527684 21333 raft_consensus.cc:2388] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
W20250116 02:59:13.494796 22591 consensus_meta.cc:287] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.815s user 0.011s sys 0.002s
I20250116 02:59:13.528129 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:13.534742 22621 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
W20250116 02:59:13.535921 22627 consensus_queue.cc:1159] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "17446c8dacf04461acb843547665250f" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20250116 02:59:13.537063 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:13.538976 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:13.538475 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:13.540439 20755 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:13.485741 22347 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:13.502210 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:13.480938 22538 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:13.517953 22563 consensus_meta.cc:287] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.992s user 0.014s sys 0.000s
I20250116 02:59:13.506855 22518 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:13.551085 22518 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:13.548370 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 46711.5us
I20250116 02:59:13.550220 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.556243 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.462586 22613 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20250116 02:59:13.514588 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:13.545650 22591 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:13.584439 20723 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.582506 20757 raft_consensus.cc:2405] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
W20250116 02:59:13.586331 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 62501.3us
I20250116 02:59:13.527954 22334 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:13.544664 22617 consensus_queue.cc:260] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:13.594311 20571 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:13.547712 22601 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:13.599354 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.610772 20749 raft_consensus.cc:2388] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:13.560504 20675 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:13.583282 22612 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:13.562964 22518 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:13.611845 22612 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:13.612632 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:13.614305 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:13.615505 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 140517us
I20250116 02:59:13.619231 22516 consensus_queue.cc:1046] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:13.623992 22441 consensus_queue.cc:1035] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:13.625216 20720 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.626643 20720 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.627900 20720 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.629443 20720 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:13.629362 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:13.630402 20599 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:13.636235 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.635792 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.637035 20749 raft_consensus.cc:2371] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3. Re-sending same reply.
I20250116 02:59:13.638062 21299 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:13.640115 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:13.646405 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 67152.5us
I20250116 02:59:13.647459 20723 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:13.404431 22599 consensus_queue.cc:237] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.660079 20748 raft_consensus.cc:1235] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:13.500831 22620 consensus_queue.cc:1046] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.217s
I20250116 02:59:13.560384 22563 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:13.654495 22381 raft_consensus.cc:668] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:13.560977 22538 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.666500 20723 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:13.664321 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 46549.5us
I20250116 02:59:13.663465 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:13.677142 20683 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:13.670351 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:13.550937 22615 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:13.654505 22437 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:13.642943 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:13.546501 22347 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.660687 20647 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.669180 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.677784 20757 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:13.608062 22334 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:13.654739 21331 raft_consensus.cc:2388] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:59:13.704003 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 44146.6us
W20250116 02:59:13.704408 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 118261us
I20250116 02:59:13.705551 20749 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:13.620932 22632 raft_consensus.cc:3049] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:13.681896 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:13.706559 20683 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:13.685528 21335 raft_consensus.cc:2388] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:13.667184 22591 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:13.671428 22601 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:13.695479 20723 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:13.678453 20756 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:13.706220 22632 raft_consensus.cc:738] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:13.673316 22355 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:13.705866 20571 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:13.725365 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.726959 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:13.733315 22624 log.cc:926] Time spent T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.099s user 0.000s sys 0.002s
W20250116 02:59:13.733729 22638 log.cc:926] Time spent T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250116 02:59:13.622231 22631 consensus_queue.cc:1046] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.113s
W20250116 02:59:13.635728 22597 consensus_queue.cc:1159] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "17446c8dacf04461acb843547665250f" responder_term: 4 status { last_received { term: 3 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20250116 02:59:13.736662 22634 log.cc:926] Time spent T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20250116 02:59:13.737004 20678 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:13.738215 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:13.714632 22626 raft_consensus.cc:2758] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250116 02:59:13.740187 22601 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:13.694506 20679 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:13.715710 22633 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:13.742226 22629 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:13.729298 20749 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
W20250116 02:59:13.767230 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 51978.2us
I20250116 02:59:13.771283 20757 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:13.780287 20647 leader_election.cc:400] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Vote denied by peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647) with higher term. Message: Invalid argument: T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:13.785848 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.786128 20647 leader_election.cc:403] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250116 02:59:13.780699 20683 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:13.788803 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:13.796161 20678 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 4
I20250116 02:59:13.794780 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:13.798942 20749 raft_consensus.cc:2371] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3. Re-sending same reply.
I20250116 02:59:13.800485 20749 raft_consensus.cc:1270] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250116 02:59:13.800765 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 95116.4us
I20250116 02:59:13.796427 21333 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:13.740248 22538 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:13.762583 22346 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:13.806655 20749 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:13.768918 22591 raft_consensus.cc:2743] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:13.777535 22633 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:13.822212 20756 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:13.831122 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.802979 22628 raft_consensus.cc:2743] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:13.836197 20756 raft_consensus.cc:1270] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:13.744480 22620 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20250116 02:59:13.742326 22518 consensus_queue.cc:1035] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
W20250116 02:59:13.843735 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 214329us
I20250116 02:59:13.791865 22645 raft_consensus.cc:2798] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:13.805402 22355 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:13.860956 20571 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:13.861249 22421 consensus_queue.cc:1035] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:13.861642 22632 consensus_queue.cc:260] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:13.838481 22633 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:13.865209 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:13.814895 22601 raft_consensus.cc:695] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:13.803205 22614 consensus_queue.cc:1035] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20250116 02:59:13.871593 20755 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:13.947376 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 264016us
I20250116 02:59:13.948549 20572 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:13.864135 22334 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:13.932101 20720 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:13.938800 20678 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250116 02:59:14.234560 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 467254us
I20250116 02:59:13.835657 22346 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:14.066897 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 251548us
I20250116 02:59:13.943593 22381 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:14.105026 22347 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.119305 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:13.943792 22645 raft_consensus.cc:695] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:14.053767 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 93208.7us
I20250116 02:59:14.130859 22601 consensus_queue.cc:237] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.094347 21327 raft_consensus.cc:1235] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:14.134282 22620 raft_consensus.cc:1059] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: attempting to promote NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 to VOTER
W20250116 02:59:14.062034 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 201137us
I20250116 02:59:14.284909 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:13.950177 22615 raft_consensus.cc:2743] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.287377 22615 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:14.075919 22632 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:14.121999 22653 raft_consensus.cc:2743] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.104277 20679 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:59:13.939818 22355 raft_consensus.cc:695] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:14.290165 22355 consensus_queue.cc:237] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:14.293258 20571 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:13.931414 22534 consensus_meta.cc:287] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.136s user 0.015s sys 0.000s
I20250116 02:59:14.057030 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 446) took 1625 ms. Trace:
I20250116 02:59:14.232687 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:14.068084 22563 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:14.074316 20680 consensus_meta.cc:287] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.607s user 0.008s sys 0.002s
I20250116 02:59:14.246270 22628 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:13.917655 22543 consensus_queue.cc:1035] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.155s
W20250116 02:59:14.123167 22630 log.cc:926] Time spent T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20250116 02:59:14.119995 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:13.930963 22636 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:14.235991 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 303985us
I20250116 02:59:14.237393 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:14.238679 22334 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.326639 20755 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:14.327095 20720 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.243257 22646 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647) with higher term. Message: Invalid argument: T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:14.243916 22566 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:14.247704 20753 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:14.225123 22565 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:14.243261 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:14.255466 20572 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:14.240113 22437 consensus_meta.cc:287] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.541s user 0.004s sys 0.006s
I20250116 02:59:14.341992 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:14.341826 22645 consensus_queue.cc:237] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:14.257148 20647 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:14.107260 22649 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:14.294677 21335 rpcz_store.cc:276] 0116 02:59:12.431304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:12.494735 (+ 63431us) service_pool.cc:225] Handling call
0116 02:59:14.056717 (+1561982us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54005933d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:14.056937 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2590903680}
I20250116 02:59:14.306803 20680 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
W20250116 02:59:13.998531 20604 consensus_meta.cc:287] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.659s user 0.011s sys 0.000s
I20250116 02:59:14.321000 20603 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:14.329766 21300 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:14.305461 22653 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:14.334406 20720 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:14.267138 22381 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.338059 22262 raft_consensus.cc:3049] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:14.294801 22347 consensus_queue.cc:1046] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20250116 02:59:14.339473 22652 raft_consensus.cc:2743] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:14.339917 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 84572.4us
I20250116 02:59:14.345861 20604 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:14.373170 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1848) took 1204 ms. Trace:
I20250116 02:59:14.373608 20604 rpcz_store.cc:276] 0116 02:59:13.169132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.169307 (+ 175us) service_pool.cc:225] Handling call
0116 02:59:13.169686 (+ 379us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:14.372776 (+1203090us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:14.373088 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:14.294559 22328 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.352442 22437 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:14.338536 22659 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:14.294222 22534 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.352085 20720 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:14.356714 22566 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.359959 22591 raft_consensus.cc:2798] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:14.398396 22524 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:14.399014 22591 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:14.398860 22648 consensus_queue.cc:1035] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20250116 02:59:14.399484 22346 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.399274 22524 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:14.400717 22671 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20250116 02:59:14.401764 22524 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:14.371471 22653 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.402740 22653 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:14.398490 22566 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:14.402772 22658 log.cc:926] Time spent T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20250116 02:59:14.383666 22328 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.350967 22628 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20250116 02:59:14.405280 22566 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.406112 22328 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.406415 22628 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.406736 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:14.378185 22417 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.408207 22591 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:14.335144 22334 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.412465 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:14.413302 20755 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
W20250116 02:59:14.414000 22546 consensus_meta.cc:287] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.896s user 0.009s sys 0.001s
I20250116 02:59:14.414660 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:14.396673 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 45075.8us
I20250116 02:59:14.356834 22262 raft_consensus.cc:738] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:14.367245 22563 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:14.417943 21300 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:14.420162 20753 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:14.395376 20572 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:14.417866 21333 raft_consensus.cc:1235] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
W20250116 02:59:14.423208 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 166150us
I20250116 02:59:14.424656 22381 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:14.370877 22413 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20250116 02:59:14.425698 20647 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:14.415493 22659 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.419464 22563 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:14.398564 22660 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20250116 02:59:14.428193 22262 consensus_queue.cc:260] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.431181 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:14.419008 22612 consensus_meta.cc:287] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.807s user 0.007s sys 0.007s
I20250116 02:59:14.350184 22636 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.436754 21335 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:14.440963 20676 raft_consensus.cc:2363] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4.
I20250116 02:59:14.442268 20755 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
W20250116 02:59:14.462450 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 83375.4us
I20250116 02:59:14.449384 21300 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:14.463605 20572 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:14.417215 22668 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:14.440891 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 98997.2us
I20250116 02:59:14.412487 22334 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.469276 21300 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:14.439321 22532 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:14.471963 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 46342.6us
I20250116 02:59:14.467471 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 673) took 1661 ms. Trace:
I20250116 02:59:14.470752 20720 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 02:59:14.507210 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 43704.4us
I20250116 02:59:14.414520 22546 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.519424 20571 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:14.447161 22563 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:14.446965 22347 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:14.536921 20647 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.464179 22413 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:14.616654 20572 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:14.467101 22612 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.620649 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:14.636498 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 117169us
I20250116 02:59:14.637439 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:14.470332 22543 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:14.519116 22622 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.625564 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:14.629633 21299 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:14.624416 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 87629.2us
I20250116 02:59:14.468556 22328 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.623152 22347 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:14.470397 22381 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:14.616340 22623 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.593943 22649 raft_consensus.cc:695] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:14.515219 22346 raft_consensus.cc:2798] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:14.623049 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:14.640400 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 47592.3us
I20250116 02:59:14.658215 20572 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:14.622442 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:14.510144 22566 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.505462 22530 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250116 02:59:14.643096 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 172537us
I20250116 02:59:14.525524 22636 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:14.648991 20647 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.520967 22659 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:14.595028 22355 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.534760 22534 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:14.517846 22664 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:14.515090 22262 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:14.690586 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:14.518222 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:14.656335 20678 raft_consensus.cc:1270] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250116 02:59:14.695374 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:14.658529 20680 raft_consensus.cc:1270] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250116 02:59:14.682739 22672 log.cc:926] Time spent T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.267s user 0.000s sys 0.001s
I20250116 02:59:14.509912 22306 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.147s
I20250116 02:59:14.508626 22668 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:14.656704 22347 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:14.679617 20647 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.656811 22381 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:14.543972 22532 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:14.716014 22532 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:14.716696 22532 consensus_queue.cc:1035] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:59:14.682855 22675 raft_consensus.cc:2758] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250116 02:59:14.683420 22667 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:14.717641 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 415) took 1362 ms. Trace:
I20250116 02:59:14.684340 22663 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:14.718868 22663 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:14.671473 22566 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.718351 22675 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:14.719866 22673 log.cc:926] Time spent T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
I20250116 02:59:14.719622 22675 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.720919 22566 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.493119 22628 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:14.722016 22675 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:14.511083 21329 rpcz_store.cc:276] 0116 02:59:12.802948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:12.821220 (+ 18272us) service_pool.cc:225] Handling call
0116 02:59:12.897439 (+ 76219us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:13.011764 (+114325us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:14.422019 (+1410255us) spinlock_profiling.cc:243] Waited 93.4 ms on lock 0x7b540060f4d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:14.463960 (+ 41941us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":214931712}
I20250116 02:59:14.689666 22530 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:14.635771 22413 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.725791 22413 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.657538 22346 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:14.688649 20753 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
W20250116 02:59:14.651207 22642 thread.cc:635] apply [worker] (thread pool) Time spent creating pthread: real 0.787s user 0.015s sys 0.007s
I20250116 02:59:14.685407 22355 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.699805 22611 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20250116 02:59:14.672513 22652 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.681627 22636 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.684037 22679 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:14.656646 22328 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.710398 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:14.752436 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:14.758344 20753 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:14.735484 22346 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:14.713554 22381 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:14.671483 20679 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:14.723130 22628 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:14.729470 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 49942.4us
I20250116 02:59:14.769307 20647 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.663681 22645 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:14.731462 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:14.718346 22678 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:14.676906 22612 consensus_queue.cc:1046] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250116 02:59:14.703639 22664 raft_consensus.cc:695] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:14.738433 22546 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.740831 20755 raft_consensus.cc:1235] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:14.704144 22659 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:14.748183 21335 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250116 02:59:14.667002 22623 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:14.718187 21327 rpcz_store.cc:276] 0116 02:59:13.354860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.359683 (+ 4823us) service_pool.cc:225] Handling call
0116 02:59:14.065883 (+706200us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:14.260771 (+194888us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:14.717408 (+456637us) spinlock_profiling.cc:243] Waited 689 ms on lock 0x7b54005933d0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:14.717568 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1585230208}
I20250116 02:59:14.762821 21300 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:14.756292 22636 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:14.734949 22632 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.732954 22663 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:14.751381 21328 raft_consensus.cc:2388] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:14.692623 22543 raft_consensus.cc:695] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:14.769752 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 132405us
I20250116 02:59:14.749480 22421 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.776293 20756 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:14.776513 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:14.828863 20603 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:14.777473 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:14.778483 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:14.784224 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:14.787161 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:14.810456 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:14.828313 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 43884us
W20250116 02:59:14.735363 22642 thread.cc:601] apply [worker] (thread pool) Time spent starting thread: real 0.872s user 0.016s sys 0.008s
I20250116 02:59:14.814013 20606 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250116 02:59:14.813164 22610 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s user 0.021s sys 0.046s
I20250116 02:59:14.814239 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:14.843708 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:14.845041 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:14.823503 20572 leader_election.cc:336] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.622s (SENT)
W20250116 02:59:14.824769 22682 log.cc:926] Time spent T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20250116 02:59:14.785627 20754 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:14.730276 22601 consensus_queue.cc:1035] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
W20250116 02:59:14.832947 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 70261.9us
I20250116 02:59:14.782716 22597 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:14.789228 22633 consensus_meta.cc:287] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.829s user 0.009s sys 0.004s
I20250116 02:59:14.828857 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:14.813881 22645 raft_consensus.cc:695] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:14.764887 22652 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.825673 22665 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20250116 02:59:14.761775 22534 consensus_queue.cc:1046] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20250116 02:59:14.840803 22543 consensus_queue.cc:237] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.842725 20607 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:14.839201 22664 consensus_queue.cc:237] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:14.846359 21302 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.814481 22530 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:14.790766 20647 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.774768 22679 raft_consensus.cc:695] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:14.849665 22334 raft_consensus.cc:2798] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:14.854867 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:14.812356 22659 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:14.855363 20723 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.849309 22689 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.856252 22687 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:14.843154 22421 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:14.857750 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:14.858400 20571 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
W20250116 02:59:14.844101 22610 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s user 0.022s sys 0.046s
I20250116 02:59:14.860867 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:14.861790 22608 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20250116 02:59:14.860774 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:14.843106 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:14.752492 22649 consensus_queue.cc:237] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:14.852380 22667 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:14.908587 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 62320.7us
W20250116 02:59:14.914028 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 62528.5us
I20250116 02:59:14.880162 22594 consensus_queue.cc:1046] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
W20250116 02:59:14.919548 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 64328.7us
I20250116 02:59:14.875795 20599 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:14.872682 22633 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:14.914960 22530 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:14.926386 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 135775us
I20250116 02:59:14.929688 21300 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:14.938067 20647 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:14.789477 22620 consensus_queue.cc:237] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:14.935987 22681 log.cc:926] Time spent T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20250116 02:59:14.916612 22334 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:14.919926 22659 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:14.939631 20608 raft_consensus.cc:1270] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:14.924160 22328 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.944587 21302 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:14.873615 22597 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:14.947924 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:14.948354 20723 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:14.925686 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 223599us
I20250116 02:59:14.924475 22601 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.944658 22677 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.975884 22606 consensus_queue.cc:1046] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:14.976274 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:14.977236 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:14.977737 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:14.977886 20679 raft_consensus.cc:1270] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:14.970018 20608 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:14.979727 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:14.983654 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 45687.5us
I20250116 02:59:14.984489 20647 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.938860 22530 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:14.989198 20720 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:14.990078 20647 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.990701 20720 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.992830 22248 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:14.993816 20720 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:14.996776 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 682) took 1347 ms. Trace:
W20250116 02:59:15.000629 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 58535.7us
I20250116 02:59:14.943765 22546 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:14.947299 22334 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:14.943288 22355 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:14.947440 22619 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:14.969081 22421 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.939134 22668 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:14.966691 22633 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:15.015727 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 188731us
I20250116 02:59:14.997318 21332 rpcz_store.cc:276] 0116 02:59:13.649295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.649596 (+ 301us) service_pool.cc:225] Handling call
0116 02:59:13.650052 (+ 456us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:13.729029 (+ 78977us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:13.876326 (+147297us) log.cc:844] Serialized 14 byte log entry
0116 02:59:14.212746 (+336420us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:14.212966 (+ 220us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:14.213143 (+ 177us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:14.996404 (+783261us) spinlock_profiling.cc:243] Waited 78.7 ms on lock 0x7b540058eb54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:14.996471 (+ 67us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:14.996670 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":130956,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":310,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":122,"spinlock_wait_cycles":180899200,"thread_start_us":434688,"threads_started":2,"wal-append.queue_time_us":367472}
W20250116 02:59:15.017020 22680 log.cc:926] Time spent T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20250116 02:59:14.967336 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:14.981717 22612 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:15.031911 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:15.032755 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:15.000211 22613 raft_consensus.cc:2798] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:15.008281 22678 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:15.000247 22328 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250116 02:59:15.037545 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 47557.2us
W20250116 02:59:15.038504 22691 log.cc:926] Time spent T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250116 02:59:15.039222 22686 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:15.010252 22636 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:14.956017 22381 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:15.030839 22591 consensus_meta.cc:287] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.622s user 0.010s sys 0.003s
I20250116 02:59:15.014539 22677 raft_consensus.cc:2798] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:15.028712 22355 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:15.036301 22546 raft_consensus.cc:2798] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:15.063542 20603 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:14.923910 22687 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.001926 22565 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20250116 02:59:15.022038 22601 consensus_queue.cc:1035] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20250116 02:59:15.080224 20604 raft_consensus.cc:1235] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:15.045092 22668 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:14.987838 22645 consensus_queue.cc:237] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.046348 22413 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:15.031096 22619 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.078423 22546 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:15.051883 22328 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:15.086185 21332 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:15.065966 22677 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:15.064462 22686 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:15.070299 22623 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:14.945782 22689 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.068876 22613 raft_consensus.cc:695] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:15.070281 22636 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:15.085906 22694 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:15.087172 22693 log.cc:926] Time spent T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20250116 02:59:14.950299 22667 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.070154 22633 consensus_queue.cc:1035] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20250116 02:59:15.069307 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:15.106786 22546 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:15.106557 22328 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:15.086743 22652 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:15.084415 22612 consensus_queue.cc:1035] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250116 02:59:15.015391 22679 consensus_queue.cc:237] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.109709 22686 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:15.108783 22677 raft_consensus.cc:3054] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:15.156392 22695 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:15.069638 22591 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.176810 22695 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.183512 22695 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:15.106937 22413 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.116501 22704 consensus_queue.cc:1046] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.146s
I20250116 02:59:15.107896 22346 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.201264 22413 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:15.201877 22597 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.186781 22518 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:15.205037 22619 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:15.190690 22563 consensus_meta.cc:287] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.555s user 0.010s sys 0.003s
I20250116 02:59:15.203754 22591 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:15.091295 22678 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.227098 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:15.213110 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:15.238871 22591 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:15.243472 22591 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.244056 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:15.244510 22678 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.246759 20602 raft_consensus.cc:2388] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:15.246842 22678 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:15.253937 22346 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.233656 22381 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.229764 22597 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:15.255340 22597 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:15.255923 20676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1437) took 1782 ms (client timeout 1603 ms). Trace:
W20250116 02:59:15.256350 20676 rpcz_store.cc:269] 0116 02:59:13.473240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.488476 (+ 15236us) service_pool.cc:225] Handling call
0116 02:59:15.255665 (+1767189us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b540016e4d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:15.255839 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2187654016}
W20250116 02:59:15.240542 22347 consensus_meta.cc:287] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.527s user 0.011s sys 0.002s
I20250116 02:59:15.270277 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:15.223081 22563 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.273751 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 1857) took 1853 ms. Trace:
I20250116 02:59:15.272665 22597 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.277784 22240 raft_consensus.cc:3049] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:15.254323 22678 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:15.221289 22518 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.294034 22667 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:15.300877 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:15.215989 22613 consensus_queue.cc:237] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.292430 22240 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:15.308663 22700 log.cc:926] Time spent T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20250116 02:59:15.273144 22347 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:15.308679 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 330382us
I20250116 02:59:15.290522 20602 rpcz_store.cc:276] 0116 02:59:13.416021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.438818 (+ 22797us) service_pool.cc:225] Handling call
0116 02:59:15.243831 (+1805013us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:15.263992 (+ 20161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26446080}
I20250116 02:59:15.316401 22563 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.277801 22328 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.321205 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:15.326110 21333 raft_consensus.cc:2388] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:15.301992 22687 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:15.307600 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:15.328306 22600 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:15.315724 22689 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.322655 22563 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:15.321637 22591 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:15.343338 22602 raft_consensus.cc:3049] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:15.345968 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 680) took 1764 ms. Trace:
I20250116 02:59:15.345201 22552 raft_consensus.cc:3049] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:15.348047 21334 rpcz_store.cc:276] 0116 02:59:13.581438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.581616 (+ 178us) service_pool.cc:225] Handling call
0116 02:59:13.597229 (+ 15613us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:13.727390 (+130161us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:13.727425 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:13.727467 (+ 42us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:13.727499 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:15.345686 (+1618187us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54004a06d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:15.345743 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:15.345896 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":298747648}
I20250116 02:59:15.307170 22678 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.345693 22616 raft_consensus.cc:3049] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Stepping down as leader of term 1
W20250116 02:59:15.368224 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 423701us
I20250116 02:59:15.350158 22552 raft_consensus.cc:738] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:15.373270 22591 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250116 02:59:15.349540 22563 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.357069 22602 raft_consensus.cc:738] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:15.380105 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:15.381381 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1860) took 1964 ms. Trace:
I20250116 02:59:15.384292 20605 rpcz_store.cc:276] 0116 02:59:13.416707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.422777 (+ 6070us) service_pool.cc:225] Handling call
0116 02:59:13.423099 (+ 322us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:13.469169 (+ 46070us) raft_consensus.cc:1529] Early marking committed up to index 4
0116 02:59:13.469198 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:13.469234 (+ 36us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:13.469267 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:15.381029 (+1911762us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x7b54002d8a54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:15.381094 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:15.381289 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":105279872}
I20250116 02:59:15.365677 20723 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:15.345427 22546 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.368532 22328 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:15.398295 20682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1427) took 1950 ms (client timeout 1704 ms). Trace:
I20250116 02:59:15.398500 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:15.400846 20682 rpcz_store.cc:269] 0116 02:59:13.448852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.448988 (+ 136us) service_pool.cc:225] Handling call
0116 02:59:15.379851 (+1930863us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:15.398207 (+ 18356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12634368}
I20250116 02:59:15.401404 20676 raft_consensus.cc:2405] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20250116 02:59:15.403272 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:15.404136 20676 raft_consensus.cc:2388] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:15.339326 22240 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.374743 22616 raft_consensus.cc:738] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:15.405751 21302 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
W20250116 02:59:15.412851 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 483205us
I20250116 02:59:15.414021 21300 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:15.393968 22591 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:15.444859 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:15.381090 22530 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.447851 20756 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:15.450526 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:15.474437 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:15.475558 20756 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:15.417656 22667 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:15.414525 22687 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:15.414520 22328 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:15.431736 22663 consensus_meta.cc:287] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.590s user 0.014s sys 0.000s
I20250116 02:59:15.421114 22636 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:15.510524 21300 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:15.414357 22622 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:15.487545 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 454854us
I20250116 02:59:15.460345 22240 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:15.497543 20756 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:15.459183 22660 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:15.518747 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:15.519652 20755 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:15.521026 21300 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:15.419723 22563 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.513229 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:15.502887 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:15.513273 20571 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:15.516690 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:15.461498 22552 consensus_queue.cc:260] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:15.525149 20607 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.005s sys 0.029s
I20250116 02:59:15.469053 22347 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.528437 20605 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:15.528721 20607 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.523s user 0.005s sys 0.029s
I20250116 02:59:15.534744 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:15.535602 20756 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:15.537082 20750 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:15.537228 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:15.537956 20753 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:15.540470 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1421) took 2062 ms. Trace:
I20250116 02:59:15.541242 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:15.550287 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:15.540938 20750 rpcz_store.cc:276] 0116 02:59:13.478004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.478172 (+ 168us) service_pool.cc:225] Handling call
0116 02:59:13.494961 (+ 16789us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:15.536904 (+2041943us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b540024b920. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326c9f990 00007fc326d11300 00007fc326d12371 00007fc326d0eedf 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc
0116 02:59:15.540160 (+ 3256us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:15.540407 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84754176}
I20250116 02:59:15.556519 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:15.558049 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:15.558959 20608 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:59:15.450452 22334 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.522050 22667 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:15.564074 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:15.564698 22347 raft_consensus.cc:2743] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:15.516088 22687 raft_consensus.cc:695] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:15.564595 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:15.587112 20750 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:15.561427 22664 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:15.521703 22636 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.565438 22667 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:15.523026 22328 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.518795 22602 consensus_queue.cc:260] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.627305 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1420) took 2149 ms. Trace:
I20250116 02:59:15.627879 20748 rpcz_store.cc:276] 0116 02:59:13.477720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.479538 (+ 1818us) service_pool.cc:225] Handling call
0116 02:59:13.500497 (+ 20959us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:13.665132 (+164635us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:15.626895 (+1961763us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b5400003a54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:15.627191 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":334782208}
I20250116 02:59:15.515806 22663 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.566745 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:15.675222 20755 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:15.675969 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 162792us
I20250116 02:59:15.722523 20571 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:15.481844 22616 consensus_queue.cc:260] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.741770 22616 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:15.540874 22622 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:15.751077 22622 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:15.620373 22667 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:15.628041 22602 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:15.560712 22660 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:15.685158 22678 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:15.855820 20753 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:15.824797 22530 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.893502 22679 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:15.868503 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 191768us
W20250116 02:59:15.763242 20720 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.027s sys 0.058s
W20250116 02:59:15.990862 20720 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.992s user 0.028s sys 0.063s
W20250116 02:59:15.991283 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 992962us
I20250116 02:59:15.822047 22664 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250116 02:59:15.866534 22421 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.009s sys 0.021s
I20250116 02:59:15.877717 20756 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:15.814669 20647 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.014s sys 0.046s
W20250116 02:59:15.761107 21331 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.889s user 0.016s sys 0.034s
I20250116 02:59:15.905316 22334 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.753917 22546 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:15.920099 20755 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:15.901191 22636 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:15.901726 22543 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.012s sys 0.026s
I20250116 02:59:15.820788 22687 consensus_queue.cc:237] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:15.972581 22640 log.cc:926] Time spent T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 2.110s user 0.000s sys 0.006s
W20250116 02:59:15.978451 22659 consensus_meta.cc:287] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.993s user 0.015s sys 0.010s
W20250116 02:59:15.980954 22721 log.cc:926] Time spent T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20250116 02:59:16.004374 20647 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.937s user 0.019s sys 0.049s
I20250116 02:59:15.997681 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:16.005486 20571 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:16.006222 20750 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:15.905776 20677 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.844s user 0.027s sys 0.012s
W20250116 02:59:15.924908 22645 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.016s sys 0.008s
W20250116 02:59:16.057170 20574 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.025s sys 0.037s
I20250116 02:59:16.772534 20676 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:16.772073 20677 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.710s user 0.046s sys 0.037s
W20250116 02:59:16.401548 21299 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.011s sys 0.016s
I20250116 02:59:16.038920 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:16.773696 21299 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.995s user 0.014s sys 0.026s
I20250116 02:59:16.774073 20682 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
W20250116 02:59:16.773166 20574 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.732s user 0.035s sys 0.072s
W20250116 02:59:16.183449 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.012s sys 0.041s
W20250116 02:59:16.775489 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.253s user 0.012s sys 0.061s
W20250116 02:59:16.776129 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 1.25509e+06us
W20250116 02:59:16.774156 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 995767us
I20250116 02:59:16.778234 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:16.005072 22421 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s user 0.010s sys 0.021s
I20250116 02:59:16.770581 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:16.799046 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:16.780855 20606 raft_consensus.cc:1270] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250116 02:59:16.811774 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1564) took 2031 ms. Trace:
I20250116 02:59:16.812060 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 346) took 1325 ms. Trace:
W20250116 02:59:16.007382 21331 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 2.135s user 0.020s sys 0.044s
W20250116 02:59:16.020797 20674 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.012s sys 0.025s
W20250116 02:59:16.778277 22613 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.933s user 0.003s sys 0.011s
W20250116 02:59:16.364151 22709 log.cc:926] Time spent T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8: Append to log took a long time: real 0.519s user 0.000s sys 0.002s
W20250116 02:59:16.079504 22686 consensus_meta.cc:287] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.883s user 0.006s sys 0.018s
W20250116 02:59:16.761518 22724 log.cc:926] Time spent T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.215s user 0.000s sys 0.000s
I20250116 02:59:16.882515 20750 rpcz_store.cc:276] 0116 02:59:15.446992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.564271 (+117279us) service_pool.cc:225] Handling call
0116 02:59:16.771990 (+1207719us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:16.056133 22328 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:16.537076 22620 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.347s user 0.024s sys 0.048s
W20250116 02:59:16.038702 21330 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.934s user 0.009s sys 0.042s
I20250116 02:59:16.882656 20606 rpcz_store.cc:276] 0116 02:59:14.780420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:14.924324 (+143904us) service_pool.cc:225] Handling call
0116 02:59:16.780231 (+1855907us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98d0e 0000000000353db9 00007fc32bbf8a70 00007fc32bcbe834 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:16.780459 (+ 228us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:16.811353 (+ 30894us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:16.811688 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5168384}
I20250116 02:59:16.889791 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:16.788527 22543 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.732s user 0.032s sys 0.050s
W20250116 02:59:16.787516 22645 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.470s user 0.033s sys 0.034s
I20250116 02:59:16.901516 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:16.277431 22604 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.883s user 0.007s sys 0.047s
I20250116 02:59:16.137490 22334 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:16.625722 22649 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:16.064999 22663 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:16.782332 22622 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:16.007021 22679 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:17.637977 20647 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.010s sys 0.013s
I20250116 02:59:17.436800 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:16.452520 21328 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250116 02:59:17.451391 20751 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.526s user 0.015s sys 0.062s
W20250116 02:59:17.703830 21299 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.010s sys 0.001s
W20250116 02:59:17.435794 20646 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.320s user 0.023s sys 0.047s
I20250116 02:59:16.955642 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:17.005333 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:17.666563 20720 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.659s user 0.043s sys 0.037s
I20250116 02:59:16.957981 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:17.758796 20647 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.985s user 0.011s sys 0.013s
W20250116 02:59:17.768103 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.929s user 0.003s sys 0.010s
I20250116 02:59:16.408627 21327 raft_consensus.cc:1235] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 4. Ops: []
W20250116 02:59:17.770215 20682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1456) took 1748 ms (client timeout 1514 ms). Trace:
W20250116 02:59:16.888602 20674 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.625s user 0.024s sys 0.053s
I20250116 02:59:17.759858 20605 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
W20250116 02:59:17.780031 20680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1461) took 1703 ms (client timeout 1687 ms). Trace:
W20250116 02:59:17.787189 20680 rpcz_store.cc:269] 0116 02:59:16.075810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:16.809356 (+733546us) service_pool.cc:225] Handling call
0116 02:59:17.779023 (+969667us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:17.753163 20573 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.971s user 0.004s sys 0.009s
W20250116 02:59:17.772529 20646 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.656s user 0.030s sys 0.049s
W20250116 02:59:17.788306 20573 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.006s user 0.004s sys 0.010s
W20250116 02:59:17.788434 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 1.67265e+06us
W20250116 02:59:17.760615 21330 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 2.657s user 0.027s sys 0.101s
W20250116 02:59:17.770411 22696 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.002s sys 0.008s
W20250116 02:59:17.637320 22609 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.569s user 0.005s sys 0.062s
W20250116 02:59:17.478466 22240 consensus_meta.cc:287] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.953s user 0.001s sys 0.020s
W20250116 02:59:17.772085 21299 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.919s user 0.010s sys 0.002s
I20250116 02:59:16.940194 22711 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:17.073908 22355 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:17.070274 22618 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:17.843047 20606 raft_consensus.cc:1235] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:17.843914 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1570) took 2326 ms. Trace:
I20250116 02:59:17.842392 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:17.844343 20606 rpcz_store.cc:276] 0116 02:59:15.517417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:17.779116 (+2261699us) service_pool.cc:225] Handling call
0116 02:59:17.842824 (+ 63708us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:17.843580 (+ 756us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:17.843854 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:17.773998 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.935s user 0.003s sys 0.010s
I20250116 02:59:17.543143 22728 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:17.780565 20682 rpcz_store.cc:269] 0116 02:59:16.021938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:16.775125 (+753187us) service_pool.cc:225] Handling call
0116 02:59:17.770109 (+994984us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:17.783917 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1450) took 2795 ms. Trace:
I20250116 02:59:16.823875 22659 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:17.754909 22727 log.cc:926] Time spent T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.293s user 0.001s sys 0.000s
W20250116 02:59:17.755494 22733 log.cc:926] Time spent T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.356s user 0.000s sys 0.001s
W20250116 02:59:16.979763 22667 consensus_meta.cc:287] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.191s user 0.011s sys 0.000s
W20250116 02:59:17.098580 22677 consensus_meta.cc:287] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.884s user 0.008s sys 0.015s
W20250116 02:59:17.632871 22616 consensus_meta.cc:287] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 1.890s user 0.011s sys 0.006s
I20250116 02:59:16.497416 22715 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:17.789456 20683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 281) took 3833 ms (client timeout 1827 ms). Trace:
I20250116 02:59:17.834450 22645 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:17.869819 20723 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.097s user 0.015s sys 0.019s
W20250116 02:59:17.190281 22566 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.288s user 0.022s sys 0.041s
W20250116 02:59:17.800637 20605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 306) took 2270 ms (client timeout 1694 ms). Trace:
W20250116 02:59:17.901844 20605 rpcz_store.cc:269] 0116 02:59:15.516299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.533519 (+ 17220us) service_pool.cc:225] Handling call
0116 02:59:17.397267 (+1863748us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:17.786756 (+389489us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8839424}
I20250116 02:59:17.849371 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 431) took 1797 ms. Trace:
I20250116 02:59:17.859905 22546 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:17.215729 22600 consensus_meta.cc:287] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 1.874s user 0.015s sys 0.009s
I20250116 02:59:17.095441 22610 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:17.780462 22620 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 2.577s user 0.042s sys 0.074s
I20250116 02:59:16.969316 22660 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:17.254000 22421 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:17.908427 22546 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:17.916728 22749 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:17.916551 22421 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:17.917304 22749 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:17.838101 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.951s user 0.001s sys 0.027s
W20250116 02:59:17.918867 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.032s user 0.002s sys 0.027s
W20250116 02:59:17.906983 22717 log.cc:926] Time spent T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.313s user 0.000s sys 0.002s
I20250116 02:59:17.920531 21330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 694) took 2937 ms. Trace:
I20250116 02:59:17.862092 22715 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:17.921612 22421 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:17.861493 22677 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:17.920993 21330 rpcz_store.cc:276] 0116 02:59:14.982846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:14.983127 (+ 281us) service_pool.cc:225] Handling call
0116 02:59:15.030519 (+ 47392us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:15.047444 (+ 16925us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:15.055657 (+ 8213us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:15.099327 (+ 43670us) log.cc:844] Serialized 26 byte log entry
0116 02:59:17.850921 (+2751594us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:17.851875 (+ 954us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:17.859793 (+ 7918us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:17.920238 (+ 60445us) raft_consensus.cc:1680] finished
0116 02:59:17.920262 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:17.920434 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1971,"raft.queue_time_us":59060,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":17425,"thread_start_us":2727839,"threads_started":1,"wal-append.queue_time_us":2284724}
I20250116 02:59:17.907299 22645 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:17.924213 22645 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:17.928560 22677 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:17.929263 22346 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:17.929827 22742 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:17.930452 22742 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:17.931726 21333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 690) took 3335 ms (client timeout 1576 ms). Trace:
W20250116 02:59:17.932183 21333 rpcz_store.cc:269] 0116 02:59:14.596655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:14.596851 (+ 196us) service_pool.cc:225] Handling call
0116 02:59:17.931464 (+3334613us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54006900d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:17.931623 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1532577664}
I20250116 02:59:17.933017 22742 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:17.931674 22421 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:17.933070 22659 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:17.934145 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:17.772763 20720 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.765s user 0.045s sys 0.037s
W20250116 02:59:17.846140 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 1.84057e+06us
I20250116 02:59:17.845584 20680 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:17.005923 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:17.398420 22729 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:17.782572 22328 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:17.774298 20753 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:59:17.851522 21299 connection.cc:232] RPC call timeout handler was delayed by -1.0315422139999555s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20250116 02:59:16.902230 22613 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.046s user 0.003s sys 0.012s
W20250116 02:59:17.840476 22609 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.783s user 0.005s sys 0.065s
I20250116 02:59:17.851768 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 276) took 4402 ms. Trace:
W20250116 02:59:17.840174 22696 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s user 0.003s sys 0.008s
I20250116 02:59:17.851040 20677 rpcz_store.cc:276] 0116 02:59:14.976417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:14.978211 (+ 1794us) service_pool.cc:225] Handling call
0116 02:59:14.978526 (+ 315us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:15.014484 (+ 35958us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:15.014523 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:15.036125 (+ 21602us) log.cc:844] Serialized 26 byte log entry
0116 02:59:16.774570 (+1738445us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:16.776075 (+ 1505us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:16.776153 (+ 78us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:17.771377 (+995224us) raft_consensus.cc:1680] finished
0116 02:59:17.771401 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:17.771579 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16753,"raft.queue_time_us":21555,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":20900,"thread_start_us":1712675,"threads_started":1,"wal-append.queue_time_us":1847510}
I20250116 02:59:17.841246 22543 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:17.861392 22667 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:17.762393 22604 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 4.465s user 0.027s sys 0.058s
W20250116 02:59:17.862282 20648 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.989s user 0.003s sys 0.014s
W20250116 02:59:17.862255 20683 rpcz_store.cc:269] 0116 02:59:13.945622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.945809 (+ 187us) service_pool.cc:225] Handling call
0116 02:59:16.955386 (+3009577us) spinlock_profiling.cc:243] Waited 4.75 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:17.779023 (+823637us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10926464}
I20250116 02:59:17.865092 20606 raft_consensus.cc:1270] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:17.843559 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 429) took 1817 ms. Trace:
W20250116 02:59:17.870625 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.983s user 0.031s sys 0.034s
I20250116 02:59:17.869789 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:17.772254 20751 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.847s user 0.020s sys 0.064s
W20250116 02:59:17.871454 20723 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.099s user 0.015s sys 0.019s
I20250116 02:59:17.781013 22649 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:17.975675 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 341) took 2558 ms. Trace:
I20250116 02:59:17.862062 22616 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:17.779512 22622 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:17.781679 22334 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:17.059953 22686 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:17.884210 20649 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.989s user 0.001s sys 0.029s
I20250116 02:59:17.862069 22599 raft_consensus.cc:3049] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
W20250116 02:59:17.901340 20574 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.125s user 0.038s sys 0.026s
W20250116 02:59:17.980481 20574 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.204s user 0.039s sys 0.026s
W20250116 02:59:17.772195 20756 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.003s sys 0.010s
I20250116 02:59:17.863754 20679 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
W20250116 02:59:17.903546 22566 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 2.026s user 0.031s sys 0.052s
I20250116 02:59:17.906315 21327 rpcz_store.cc:276] 0116 02:59:16.049959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:16.051067 (+ 1108us) service_pool.cc:225] Handling call
0116 02:59:16.179755 (+128688us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:17.786483 (+1606728us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:17.847398 (+ 60915us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:17.904481 22602 consensus_meta.cc:287] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.902s user 0.011s sys 0.009s
I20250116 02:59:17.914731 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1445) took 3279 ms. Trace:
I20250116 02:59:17.852069 22355 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:17.853852 22711 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:17.934311 22421 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:17.935508 22715 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:17.770539 22662 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 2.906s user 0.000s sys 0.023s
I20250116 02:59:17.999171 22686 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:17.999838 22667 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:17.991305 22746 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:17.937091 22749 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:17.854930 22728 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:17.937870 20608 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:17.977684 22543 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:17.962582 20680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1460) took 1902 ms (client timeout 2205 ms). Trace:
W20250116 02:59:17.962826 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 1.96344e+06us
W20250116 02:59:18.007432 20680 rpcz_store.cc:269] 0116 02:59:16.060416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:17.788089 (+1727673us) service_pool.cc:225] Handling call
0116 02:59:17.962496 (+174407us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:17.980415 22616 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:17.912341 22519 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:17.966135 21299 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 2.208s (SENT) [suppressed 34 similar messages]
I20250116 02:59:17.966980 20571 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:18.177665 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 213932us
I20250116 02:59:18.178426 20571 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:18.179502 20571 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:18.179634 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1428) took 4699 ms. Trace:
I20250116 02:59:17.939126 22645 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:17.967722 20681 rpcz_store.cc:276] 0116 02:59:13.441416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.443098 (+ 1682us) service_pool.cc:225] Handling call
0116 02:59:13.446795 (+ 3697us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:13.457375 (+ 10580us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:13.457409 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:13.466962 (+ 9553us) log.cc:844] Serialized 26 byte log entry
0116 02:59:13.544014 (+ 77052us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:13.544055 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:13.544133 (+ 78us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:15.971732 (+2427599us) spinlock_profiling.cc:243] Waited 54.3 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:17.836393 (+1864661us) raft_consensus.cc:1680] finished
0116 02:59:17.836423 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:17.843925 (+ 7502us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71403,"raft.queue_time_us":43063,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":5086,"spinlock_wait_cycles":124863744,"thread_start_us":76265,"threads_started":1,"wal-append.queue_time_us":325151}
W20250116 02:59:17.972131 20648 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.200s user 0.004s sys 0.014s
I20250116 02:59:17.973301 20683 raft_consensus.cc:1270] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250116 02:59:18.180943 22645 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:18.181435 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1457) took 2144 ms. Trace:
I20250116 02:59:17.974249 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1571) took 2456 ms. Trace:
I20250116 02:59:18.181751 22645 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250116 02:59:17.974932 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.088s user 0.032s sys 0.034s
I20250116 02:59:18.182451 22645 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:18.183725 21302 connection.cc:232] RPC call timeout handler was delayed by -1.2816040950000342s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20250116 02:59:18.184190 21302 connection.cc:232] RPC call timeout handler was delayed by -1.2348040949999586s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
I20250116 02:59:17.976715 20682 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
W20250116 02:59:18.184945 20682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1459) took 2131 ms (client timeout 2785 ms). Trace:
W20250116 02:59:18.185312 20682 rpcz_store.cc:269] 0116 02:59:16.053573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:17.853186 (+1799613us) service_pool.cc:225] Handling call
0116 02:59:18.184880 (+331694us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:18.185357 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:17.979200 20649 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.084s user 0.003s sys 0.030s
I20250116 02:59:18.184965 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:18.184587 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:18.187144 20681 raft_consensus.cc:604] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:18.187379 20676 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:18.185375 20608 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1577) took 2624 ms (client timeout 2785 ms). Trace:
W20250116 02:59:18.188426 20608 rpcz_store.cc:269] 0116 02:59:15.560601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:16.957513 (+1396912us) service_pool.cc:225] Handling call
0116 02:59:18.185210 (+1227697us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b5400108454. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:18.185315 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378704256}
I20250116 02:59:18.189496 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:18.180044 20754 rpcz_store.cc:276] 0116 02:59:13.480171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.544890 (+ 64719us) service_pool.cc:225] Handling call
0116 02:59:13.610166 (+ 65276us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:14.850974 (+1240808us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.179354 (+3328380us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540033c7d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:18.179542 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2635593600}
W20250116 02:59:18.184558 21302 connection.cc:232] RPC call timeout handler was delayed by -1.0368040949999795s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1966s for a response
I20250116 02:59:18.181838 20683 rpcz_store.cc:276] 0116 02:59:16.037110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:17.972758 (+1935648us) service_pool.cc:225] Handling call
0116 02:59:17.973038 (+ 280us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:18.181125 (+208087us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.181376 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:18.191560 21302 connection.cc:232] RPC call timeout handler was delayed by -1.0275539290000779s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1942s for a response
I20250116 02:59:18.191190 22629 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.192533 22629 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:17.966434 22729 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.182068 20606 rpcz_store.cc:276] 0116 02:59:15.517696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:17.845309 (+2327613us) service_pool.cc:225] Handling call
0116 02:59:17.864636 (+ 19327us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:17.974019 (+109383us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:17.974200 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:18.194296 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:17.975986 20751 rpcz_store.cc:276] 0116 02:59:15.417091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.417740 (+ 649us) service_pool.cc:225] Handling call
0116 02:59:15.426453 (+ 8713us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:15.426812 (+ 359us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:15.925306 (+498494us) log.cc:844] Serialized 14 byte log entry
0116 02:59:17.975366 (+2050060us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:17.975424 (+ 58us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:17.975469 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:17.975519 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:17.975611 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1522216,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"thread_start_us":2509771,"threads_started":2,"wal-append.queue_time_us":2005079}
I20250116 02:59:17.929873 22346 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:17.852756 22618 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:17.982520 20756 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.034s user 0.003s sys 0.010s
I20250116 02:59:17.988114 20603 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:18.195574 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.196354 22618 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:17.975047 22604 raft_consensus.cc:695] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:18.197666 22604 consensus_queue.cc:237] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:17.988031 20678 rpcz_store.cc:276] 0116 02:59:14.637442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:14.699682 (+ 62240us) service_pool.cc:225] Handling call
0116 02:59:14.700165 (+ 483us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:14.700562 (+ 397us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:14.700590 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:15.285421 (+584831us) spinlock_profiling.cc:243] Waited 972 us on lock 0x7b2c0005b468. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc3256f0741 00007fc3256ffd72 00007fc326d4b141 00007fc326d0f58a 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782
0116 02:59:15.286019 (+ 598us) log.cc:844] Serialized 26 byte log entry
0116 02:59:15.642320 (+356301us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:15.902302 (+259982us) log.cc:844] Serialized 14 byte log entry
0116 02:59:15.902703 (+ 401us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:15.902812 (+ 109us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:17.911950 (+2009138us) raft_consensus.cc:1680] finished
0116 02:59:17.914462 (+ 2512us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:17.914648 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28026,"prepare.queue_time_us":259075,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":3392,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":14523,"spinlock_wait_cycles":2234752,"thread_start_us":355664,"threads_started":1,"wal-append.queue_time_us":992883}
I20250116 02:59:18.196381 22346 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:17.915880 22610 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.194980 22729 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:18.199329 22546 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:18.200551 22610 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:17.977149 20675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 279) took 4108 ms (client timeout 2208 ms). Trace:
I20250116 02:59:18.201095 22620 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:17.993328 20755 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:18.201723 22620 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.205880 22620 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:17.983691 22600 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:17.980226 22599 raft_consensus.cc:738] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:18.206928 22600 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.207854 22629 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:17.969755 22328 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:17.936928 20647 connection.cc:232] RPC call timeout handler was delayed by -1.4540230100000144s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15760000000000002s for a response
I20250116 02:59:18.208213 20603 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:17.987833 22647 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.209085 22600 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:18.208904 22647 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:17.997112 20753 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:18.210871 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 354) took 1208 ms. Trace:
I20250116 02:59:18.210999 22647 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:17.907771 20600 raft_consensus.cc:1235] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:17.975266 21328 rpcz_store.cc:276] 0116 02:59:16.020711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:16.037445 (+ 16734us) service_pool.cc:225] Handling call
0116 02:59:16.138678 (+101233us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:17.782824 (+1644146us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:17.838272 (+ 55448us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:18.004570 22711 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:18.026341 22616 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:17.999219 22662 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 3.260s user 0.000s sys 0.026s
I20250116 02:59:18.214478 22328 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:18.201463 20675 rpcz_store.cc:269] 0116 02:59:13.866008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.873046 (+ 7038us) service_pool.cc:225] Handling call
0116 02:59:16.482202 (+2609156us) spinlock_profiling.cc:243] Waited 4.86 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98d0e 0000000000353db9 00007fc32bbf8a70 00007fc32bcbe834 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:17.934025 (+1451823us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98dac 00007fc323d98d16 0000000000353db9 00007fc32bbf8a70 00007fc32bcbe834 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:17.969309 (+ 35284us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111047168}
W20250116 02:59:18.208495 20647 connection.cc:232] RPC call timeout handler was delayed by -1.1219230100000459s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
W20250116 02:59:18.215863 20647 connection.cc:232] RPC call timeout handler was delayed by -1.0508230100000446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
I20250116 02:59:18.220363 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1889) took 3197 ms. Trace:
I20250116 02:59:18.002436 22741 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:18.198364 20649 connection.cc:232] RPC call timeout handler was delayed by -1.0177345600000081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
W20250116 02:59:18.221365 20649 connection.cc:232] RPC call timeout handler was delayed by -1.007774626000014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2398s for a response
I20250116 02:59:18.221287 22741 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.222285 20503 catalog_manager.cc:5526] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:59:18.224483 22741 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:18.099607 22617 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:18.170846 21331 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.896s user 0.013s sys 0.037s
I20250116 02:59:18.222215 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.190824 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:18.227566 22519 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.228402 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.229012 20608 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:18.229933 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.230038 22519 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.220855 20600 rpcz_store.cc:276] 0116 02:59:15.022854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.035667 (+ 12813us) service_pool.cc:225] Handling call
0116 02:59:15.038541 (+ 2874us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:18.211572 (+3173031us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.220075 (+ 8503us) spinlock_profiling.cc:243] Waited 2.83 s on lock 0x7b54002e4fd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:18.220290 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6519972224}
I20250116 02:59:18.231688 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.233012 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:18.233727 20600 raft_consensus.cc:3054] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:18.224721 20503 catalog_manager.cc:5526] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 2 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:18.235862 20503 catalog_manager.cc:5526] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:18.017405 22679 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:18.104075 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:18.225270 21331 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.960s user 0.013s sys 0.037s
I20250116 02:59:18.243280 20607 raft_consensus.cc:2405] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20250116 02:59:18.243876 21331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 684) took 4535 ms. Trace:
W20250116 02:59:18.243781 20607 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 308) took 2726 ms (client timeout 1942 ms). Trace:
I20250116 02:59:18.238319 20755 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:18.112932 22715 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:18.113590 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.243767 22594 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:18.245524 22715 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.245635 22594 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.247918 22715 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.116227 22606 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.248704 22606 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.249383 22594 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:18.245641 20599 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
W20250116 02:59:18.250520 20599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 303) took 2745 ms (client timeout 3172 ms). Trace:
W20250116 02:59:18.116662 20572 connection.cc:232] RPC call timeout handler was delayed by -1.8576979389998769s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
I20250116 02:59:18.117192 22518 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.105180 22659 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:18.251451 20572 connection.cc:232] RPC call timeout handler was delayed by -1.1107181879999644s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
I20250116 02:59:18.251866 22659 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.254057 22659 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:18.119415 22728 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:18.005332 22746 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.254798 20606 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:18.255728 21334 raft_consensus.cc:1235] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:18.256155 22746 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:18.120035 22612 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:18.027998 22739 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.256588 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.256959 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 432) took 2202 ms. Trace:
I20250116 02:59:18.256942 22739 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.072180 22649 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:18.123569 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1561) took 3344 ms. Trace:
I20250116 02:59:18.073040 22355 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.101567 22602 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.076995 22622 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:18.144557 20723 connection.cc:232] RPC call timeout handler was delayed by -1.3009065799999462s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20250116 02:59:18.149897 22532 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.245949 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 351) took 1468 ms. Trace:
I20250116 02:59:18.259035 22532 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:18.258536 20723 connection.cc:232] RPC call timeout handler was delayed by -1.1695065799999611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.166s for a response
I20250116 02:59:18.259738 20755 rpcz_store.cc:276] 0116 02:59:16.777503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:16.777936 (+ 433us) service_pool.cc:225] Handling call
0116 02:59:18.245757 (+1467821us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540007dad4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:18.245883 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2792536960}
I20250116 02:59:18.261395 22532 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.151468 22552 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:18.245836 22636 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.263831 22636 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.251034 22606 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.155273 22619 raft_consensus.cc:3049] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:18.166592 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.258895 22612 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:18.169752 20679 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 294) took 2873 ms (client timeout 1694 ms). Trace:
I20250116 02:59:18.192828 21302 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:18.196506 20683 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:18.207435 22599 consensus_queue.cc:260] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:17.987581 22751 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.068630 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 329) took 3829 ms. Trace:
I20250116 02:59:18.073539 22334 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:18.098758 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1437) took 3426 ms. Trace:
I20250116 02:59:18.024566 22421 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:18.244482 20607 rpcz_store.cc:269] 0116 02:59:15.516868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.535057 (+ 18189us) service_pool.cc:225] Handling call
0116 02:59:18.103882 (+2568825us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:18.243703 (+139821us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8030080}
I20250116 02:59:18.211239 20753 rpcz_store.cc:276] 0116 02:59:17.002985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:17.004909 (+ 1924us) service_pool.cc:225] Handling call
0116 02:59:18.210780 (+1205871us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:18.257308 20606 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:18.144143 22601 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.075845 22543 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.259248 22602 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:18.266800 22619 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:18.278549 22619 consensus_queue.cc:260] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.279592 22619 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:18.279875 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1424) took 4800 ms (client timeout 1622 ms). Trace:
W20250116 02:59:18.267320 20602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1569) took 2750 ms (client timeout 3175 ms). Trace:
W20250116 02:59:18.280705 20602 rpcz_store.cc:269] 0116 02:59:15.517154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.166176 (+2649022us) service_pool.cc:225] Handling call
0116 02:59:18.267236 (+101060us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:18.281834 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.269490 22612 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:18.270280 22751 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.283104 20602 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:18.280972 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 352) took 1503 ms. Trace:
I20250116 02:59:18.257951 20604 rpcz_store.cc:276] 0116 02:59:14.779567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:14.920753 (+141186us) service_pool.cc:225] Handling call
0116 02:59:15.079999 (+159246us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:15.083710 (+ 3711us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.123265 (+3039555us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54001603d0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:18.123448 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":365434240}
I20250116 02:59:18.284346 22518 consensus_queue.cc:1046] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:59:18.285609 22751 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.285732 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.286510 20604 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:18.287413 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.244288 21331 rpcz_store.cc:276] 0116 02:59:13.708764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.711572 (+ 2808us) service_pool.cc:225] Handling call
0116 02:59:13.763200 (+ 51628us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:13.796333 (+ 33133us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:15.294301 (+1497968us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc323db881e 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd3305 00007fc32a6e4107
0116 02:59:17.146810 (+1852509us) log.cc:844] Serialized 14 byte log entry
0116 02:59:17.903131 (+756321us) spinlock_profiling.cc:243] Waited 5.51 ms on lock 0x7b18001a5a80. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd67e8 00007fc323dd9cc1 00007fc323dd9c79 00007fc323dd9c09 00007fc323dd9bd1 00007fc323dd8ecd 00007fc32ce13234 00007fc32ce13069 00007fc323db8fe6
0116 02:59:18.243481 (+340350us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:18.243549 (+ 68us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:18.243599 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.243680 (+ 81us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:18.243805 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":45504,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":122,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":105429120,"thread_start_us":4018632,"threads_started":2,"wal-append.queue_time_us":988498}
I20250116 02:59:18.288599 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:18.289207 20604 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:18.270341 22599 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:59:18.274847 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 82109.4us
I20250116 02:59:18.285065 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.257488 21334 rpcz_store.cc:276] 0116 02:59:16.054150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:16.054387 (+ 237us) service_pool.cc:225] Handling call
0116 02:59:16.057005 (+ 2618us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:18.256184 (+2199179us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.256718 (+ 534us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54005933d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:18.256900 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5055860992}
I20250116 02:59:18.272810 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:18.271895 20723 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.270694 20749 rpcz_store.cc:276] 0116 02:59:14.239323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:14.239629 (+ 306us) service_pool.cc:225] Handling call
0116 02:59:14.240172 (+ 543us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:14.506731 (+266559us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:14.506769 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:14.506815 (+ 46us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:14.506847 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.068331 (+3561484us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54001d1854. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:18.068387 (+ 56us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:18.068548 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":612439808}
I20250116 02:59:18.272141 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:18.250944 20599 rpcz_store.cc:269] 0116 02:59:15.504719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.508157 (+ 3438us) service_pool.cc:225] Handling call
0116 02:59:18.113439 (+2605282us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98d0e 0000000000353db9 00007fc32bbf8a70 00007fc32bcbe834 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:18.250449 (+137010us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5787392}
I20250116 02:59:18.324432 20607 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:18.325409 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.271204 20757 rpcz_store.cc:276] 0116 02:59:14.672807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:14.672950 (+ 143us) service_pool.cc:225] Handling call
0116 02:59:14.683823 (+ 10873us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:14.684339 (+ 516us) raft_consensus.cc:1529] Early marking committed up to index 2
0116 02:59:14.684700 (+ 361us) log.cc:844] Serialized 14 byte log entry
0116 02:59:14.865394 (+180694us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:14.875204 (+ 9810us) log.cc:844] Serialized 26 byte log entry
0116 02:59:14.875416 (+ 212us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:14.875455 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:14.875545 (+ 90us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:18.061462 (+3185917us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b1800056e00. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd67e8 00007fc323dd9cc1 00007fc323dd9c79 00007fc323dd9c09 00007fc323dd9bd1 00007fc323dd8ecd 00007fc32ce13234 00007fc32ce13069 00007fc323db8fe6
0116 02:59:18.098311 (+ 36849us) raft_consensus.cc:1680] finished
0116 02:59:18.098434 (+ 123us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:18.098680 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1785,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":3816,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":12237,"spinlock_wait_cycles":3208960,"thread_start_us":180180,"threads_started":1,"wal-append.queue_time_us":193065}
I20250116 02:59:18.269888 22739 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:18.327064 20599 raft_consensus.cc:2388] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:18.266579 22636 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.328317 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:18.324520 20602 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
W20250116 02:59:18.280304 20752 rpcz_store.cc:269] 0116 02:59:13.478949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.490513 (+ 11564us) service_pool.cc:225] Handling call
0116 02:59:18.279613 (+4789100us) spinlock_profiling.cc:243] Waited 4.75 s on lock 0x7b5400185954. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:18.279799 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10928304896}
I20250116 02:59:18.243317 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.330526 21300 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:18.291185 20756 rpcz_store.cc:276] 0116 02:59:16.777902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:16.778131 (+ 229us) service_pool.cc:225] Handling call
0116 02:59:16.778427 (+ 296us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:16.778736 (+ 309us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:18.195392 (+1416656us) log.cc:844] Serialized 14 byte log entry
0116 02:59:18.280622 (+ 85230us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:18.280683 (+ 61us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:18.280734 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.280800 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:18.280913 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":136945,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":43,"raft.queue_time_us":3166,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":111,"thread_start_us":1331388,"threads_started":2,"wal-append.queue_time_us":84928}
I20250116 02:59:18.325707 20606 raft_consensus.cc:3049] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:18.332404 20606 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:18.277347 20676 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
W20250116 02:59:18.335510 20676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1455) took 2330 ms (client timeout 1660 ms). Trace:
W20250116 02:59:18.336007 20676 rpcz_store.cc:269] 0116 02:59:16.004854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:16.776525 (+771671us) service_pool.cc:225] Handling call
0116 02:59:18.184412 (+1407887us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:18.335410 (+150998us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53285760}
W20250116 02:59:18.291524 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 325378us
I20250116 02:59:18.330741 20608 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:18.352483 20505 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41972 (request call id 335) took 6980 ms. Trace:
I20250116 02:59:18.285890 22518 raft_consensus.cc:3049] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:18.333276 20606 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:18.356052 20603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 304) took 2851 ms (client timeout 1746 ms). Trace:
I20250116 02:59:18.356678 20649 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.358357 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:18.360993 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.363221 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.329658 21302 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:18.391762 20674 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.006s sys 0.011s
W20250116 02:59:18.407174 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 135370us
W20250116 02:59:18.407224 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 77659.7us
W20250116 02:59:18.406833 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 122639us
W20250116 02:59:18.392163 20646 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.007s sys 0.009s
I20250116 02:59:18.394800 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1458) took 2354 ms. Trace:
W20250116 02:59:18.408630 20646 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.007s sys 0.009s
I20250116 02:59:18.408799 20677 rpcz_store.cc:276] 0116 02:59:16.040677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:17.975907 (+1935230us) service_pool.cc:225] Handling call
0116 02:59:17.976276 (+ 369us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:17.984867 (+ 8591us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:17.984888 (+ 21us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:17.988706 (+ 3818us) log.cc:844] Serialized 26 byte log entry
0116 02:59:18.392727 (+404021us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:18.392779 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.392895 (+ 116us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:18.394489 (+ 1594us) raft_consensus.cc:1680] finished
0116 02:59:18.394507 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:18.394723 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3109,"raft.queue_time_us":6684,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":7823,"thread_start_us":389923,"threads_started":1,"wal-append.queue_time_us":400079}
I20250116 02:59:18.397955 21300 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:18.398663 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.398833 20606 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:18.413638 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 151646us
I20250116 02:59:18.413929 21300 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:18.415035 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.416613 21300 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 02:59:18.398933 20603 rpcz_store.cc:269] 0116 02:59:15.505039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.512882 (+ 7843us) service_pool.cc:225] Handling call
0116 02:59:18.355815 (+2842933us) spinlock_profiling.cc:243] Waited 2.46 s on lock 0x7b5400109354. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:18.355967 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5658204928}
I20250116 02:59:18.371902 21299 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:18.406015 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.418427 20604 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:18.411958 20678 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:18.423401 21300 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 02:59:18.397274 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 217811us
I20250116 02:59:18.275543 22552 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:18.420205 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.433269 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.276544 22690 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.447067 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.299057 22612 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.454298 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:18.435539 20649 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:18.274600 22748 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.441138 21302 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.277369 22602 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:18.453509 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 81709.8us
I20250116 02:59:18.355690 22334 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:18.418761 22604 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.475091 21302 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.476141 21302 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.477177 20649 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.476958 21302 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:18.405187 22532 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.465296 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.353395 20505 rpcz_store.cc:276] 0116 02:59:11.371877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:11.374818 (+ 2941us) service_pool.cc:225] Handling call
0116 02:59:18.352359 (+6977541us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:12.779056 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:12.779800 (+ 744us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:12.779866 (+ 66us) write_op.cc:435] Acquired schema lock
0116 02:59:12.779893 (+ 27us) tablet.cc:659] Decoding operations
0116 02:59:12.884433 (+104540us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:12.949286 (+ 64853us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:12.949327 (+ 41us) tablet.cc:682] Acquiring locks for 15 operations
0116 02:59:13.211963 (+262636us) tablet.cc:698] Row locks acquired
0116 02:59:13.212008 (+ 45us) write_op.cc:260] PREPARE: finished
0116 02:59:13.212200 (+ 192us) write_op.cc:270] Start()
0116 02:59:13.212358 (+ 158us) write_op.cc:276] Timestamp: P: 1736996353212127 usec, L: 0
0116 02:59:13.212395 (+ 37us) op_driver.cc:348] REPLICATION: starting
0116 02:59:13.213150 (+ 755us) log.cc:844] Serialized 6107 byte log entry
0116 02:59:13.863423 (+650273us) op_driver.cc:464] REPLICATION: finished
0116 02:59:14.535872 (+672449us) spinlock_profiling.cc:243] Waited 67.3 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc323db881e 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32a722d20
0116 02:59:14.833245 (+297373us) write_op.cc:301] APPLY: starting
0116 02:59:14.852469 (+ 19224us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:14.904495 (+ 52026us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:14.904539 (+ 44us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:14.967136 (+ 62597us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:14.967472 (+ 336us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0116 02:59:14.967537 (+ 65us) write_op.cc:312] APPLY: finished
0116 02:59:14.968068 (+ 531us) log.cc:844] Serialized 107 byte log entry
0116 02:59:15.478455 (+510387us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:15.478706 (+ 251us) write_op.cc:454] Released schema lock
0116 02:59:15.531977 (+ 53271us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:15.607121 (+ 75144us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:15.614240 (+ 7119us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":886905,"apply.run_cpu_time_us":50792,"apply.run_wall_time_us":820101,"num_ops":15,"prepare.queue_time_us":191496,"prepare.run_cpu_time_us":22529,"prepare.run_wall_time_us":694794,"raft.queue_time_us":275113,"raft.run_cpu_time_us":24926,"raft.run_wall_time_us":1014962,"replication_time_us":638375,"spinlock_wait_cycles":154816384,"thread_start_us":1960412,"threads_started":5,"wal-append.queue_time_us":1110732,"wal-append.run_cpu_time_us":35907,"wal-append.run_wall_time_us":3494599}]]}
I20250116 02:59:18.495103 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1574) took 2976 ms. Trace:
I20250116 02:59:18.439549 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1579) took 2697 ms. Trace:
I20250116 02:59:18.406136 22758 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:18.450146 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.630053 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.447732 22346 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:18.632243 22346 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:18.412722 22636 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:18.398190 22518 raft_consensus.cc:738] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:18.546126 20600 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:18.633455 22636 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.634157 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.634858 20600 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:18.473807 22748 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.635133 22518 consensus_queue.cc:260] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.636374 22518 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:18.638244 22748 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:18.486140 21299 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:18.462034 21300 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:18.439872 22565 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:18.448621 22552 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:18.421356 22728 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:18.644186 22565 raft_consensus.cc:695] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:18.644467 22728 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:18.644655 22749 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:18.644129 22552 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:18.644095 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 182130us
I20250116 02:59:18.493539 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.647130 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.648406 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.649575 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.650205 20608 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
W20250116 02:59:18.489077 20605 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.030s sys 0.038s
W20250116 02:59:18.650854 20605 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s user 0.035s sys 0.056s
I20250116 02:59:18.651669 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1573) took 3133 ms. Trace:
I20250116 02:59:18.652032 20605 rpcz_store.cc:276] 0116 02:59:15.518284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:17.902901 (+2384617us) service_pool.cc:225] Handling call
0116 02:59:17.904261 (+ 1360us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:17.904513 (+ 252us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:17.904541 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:17.907570 (+ 3029us) log.cc:844] Serialized 26 byte log entry
0116 02:59:18.651273 (+743703us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:18.651322 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.651400 (+ 78us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:18.651473 (+ 73us) raft_consensus.cc:1680] finished
0116 02:59:18.651485 (+ 12us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:18.651610 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1033984,"thread_start_us":686941,"threads_started":1,"wal-append.queue_time_us":727046}
I20250116 02:59:18.514518 20571 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:18.653429 22552 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:18.654537 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 160103us
I20250116 02:59:18.655326 22754 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:18.656023 22552 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:18.659235 22749 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:18.659775 22749 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:18.663275 21300 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:18.665112 22753 raft_consensus.cc:2798] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:18.666083 22753 raft_consensus.cc:695] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:18.461614 22612 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.667160 22753 consensus_queue.cc:237] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.668790 22612 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:18.645440 22334 raft_consensus.cc:2798] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:18.670032 22334 raft_consensus.cc:695] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:18.459352 22690 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.548481 22421 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:18.459210 22679 consensus_queue.cc:1035] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20250116 02:59:18.490028 22631 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:18.483485 22594 raft_consensus.cc:2798] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:18.518375 22766 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:18.672555 22421 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:18.268071 20679 rpcz_store.cc:269] 0116 02:59:15.296320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.297507 (+ 1187us) service_pool.cc:225] Handling call
0116 02:59:18.169468 (+2871961us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b54001c52d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:18.169640 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5788352384}
W20250116 02:59:18.407498 20674 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s user 0.006s sys 0.011s
I20250116 02:59:18.451515 22764 consensus_queue.cc:1035] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.133s
I20250116 02:59:18.673139 22421 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:18.514221 21301 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.009s sys 0.020s
W20250116 02:59:18.674186 21301 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.914s user 0.017s sys 0.035s
I20250116 02:59:18.653648 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:18.673501 20601 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:18.643344 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 157272us
I20250116 02:59:18.517822 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.680855 20602 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:18.681905 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.682641 20602 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:18.683658 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.684285 20602 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:59:18.567809 22355 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:18.685642 22355 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.687661 22748 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.517527 22747 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:18.688266 22748 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.688939 22552 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:18.689755 22552 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.420603 22755 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:18.272699 22601 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.694211 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.467010 22660 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:18.600154 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:18.490821 22696 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:18.519707 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:18.453044 22546 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:18.494091 22530 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:18.609267 22662 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.532586 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96dc04669716445ab04931358be7efbf"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:18.626571 20606 rpcz_store.cc:276] 0116 02:59:15.518573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.325157 (+2806584us) service_pool.cc:225] Handling call
0116 02:59:18.325576 (+ 419us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:18.494702 (+169126us) raft_consensus.cc:1529] Early marking committed up to index 5
0116 02:59:18.494740 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:18.494789 (+ 49us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:18.494831 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.494904 (+ 73us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:18.495036 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5560,"raft.queue_time_us":26391,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":48728,"spinlock_wait_cycles":2944}
I20250116 02:59:18.746292 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.747197 20608 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:18.747398 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.748522 20606 raft_consensus.cc:2405] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20250116 02:59:18.630851 20604 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:18.483765 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:18.493780 22532 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.632318 22758 raft_consensus.cc:695] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:18.640051 22636 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.632655 22346 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:18.490355 22604 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.627760 20599 rpcz_store.cc:276] 0116 02:59:15.733130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.327879 (+2594749us) service_pool.cc:225] Handling call
0116 02:59:18.328264 (+ 385us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:18.328725 (+ 461us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:18.328769 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:18.329286 (+ 517us) log.cc:844] Serialized 26 byte log entry
0116 02:59:18.403440 (+ 74154us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:18.403499 (+ 59us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.403618 (+ 119us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:18.404896 (+ 1278us) raft_consensus.cc:1680] finished
0116 02:59:18.404916 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:18.430108 (+ 25192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":428,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":69,"thread_start_us":72987,"threads_started":1,"wal-append.queue_time_us":73479}
I20250116 02:59:18.632788 20754 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250116 02:59:18.440145 22328 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:18.645119 22565 consensus_queue.cc:237] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.645758 22728 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:18.671137 22334 consensus_queue.cc:237] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.514927 22667 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:18.596781 20751 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:18.677307 21299 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:18.675981 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.688705 22747 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:18.459149 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.693289 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:18.694890 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.695942 20723 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.694233 20682 raft_consensus.cc:1235] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:18.697324 20676 raft_consensus.cc:1235] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:18.709295 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:18.719534 20750 raft_consensus.cc:2388] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:18.688176 22355 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:18.840984 22679 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:18.849630 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:18.850057 22679 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:18.833935 22715 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:18.852192 22667 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:18.852658 22667 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:18.850980 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:18.854651 20750 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:18.855453 20605 raft_consensus.cc:2405] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250116 02:59:18.748967 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 53041us
I20250116 02:59:18.854898 21336 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
W20250116 02:59:18.855381 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 159514us
I20250116 02:59:18.861868 20723 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:18.864428 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 187173us
I20250116 02:59:18.834738 20649 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.851837 22715 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:18.870467 20723 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.881755 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:18.891530 20757 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:18.893334 21332 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250116 02:59:18.899775 20677 raft_consensus.cc:1235] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250116 02:59:18.901476 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.902328 20603 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:18.905568 20723 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:18.689242 22594 raft_consensus.cc:695] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:18.910108 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 75543.9us
I20250116 02:59:18.702661 22755 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:18.928841 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:18.929230 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.915745 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:18.930090 20757 raft_consensus.cc:2388] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 2: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:18.933221 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:18.718760 22660 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:18.934201 20750 raft_consensus.cc:2388] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:18.747810 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.746470 22600 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:19.080221 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:19.133193 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:19.270817 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:19.335572 20751 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:18.749121 22530 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:19.300733 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:19.132200 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:19.153419 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.032s sys 0.031s
I20250116 02:59:19.154417 20601 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
W20250116 02:59:19.331260 20755 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.865s user 0.001s sys 0.039s
I20250116 02:59:19.147683 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:19.361994 20676 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:19.333024 20753 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.023s sys 0.023s
I20250116 02:59:19.127606 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:19.332119 20752 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.006s sys 0.033s
W20250116 02:59:19.363348 20752 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.831s user 0.006s sys 0.033s
I20250116 02:59:19.331945 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:18.729897 22696 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:18.857093 22441 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:18.771415 22546 raft_consensus.cc:695] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:18.930955 22599 consensus_meta.cc:287] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.508s user 0.009s sys 0.001s
I20250116 02:59:18.910506 22566 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:19.363368 20757 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:18.928129 22612 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:18.928691 22417 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:19.332196 20748 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.022s sys 0.017s
I20250116 02:59:18.914939 22764 raft_consensus.cc:2798] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:18.930392 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:18.928364 22619 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:19.099705 22740 raft_consensus.cc:2798] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:19.408087 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:19.407505 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:19.406426 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:18.952533 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 292) took 3702 ms. Trace:
I20250116 02:59:19.367209 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:19.370685 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.879s user 0.037s sys 0.035s
W20250116 02:59:19.363680 20753 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s user 0.024s sys 0.023s
W20250116 02:59:19.369985 20755 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.902s user 0.001s sys 0.039s
I20250116 02:59:19.339046 22660 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:18.894302 22747 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:18.936236 22355 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:19.418766 22612 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:19.298708 22715 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:19.375741 22441 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:19.274689 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:19.336547 22748 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:18.771790 22662 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:19.298789 22679 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:19.462479 20681 raft_consensus.cc:604] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:18.934511 22755 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:19.465436 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:19.364121 22752 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:19.368750 22696 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:19.456132 20748 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.825s user 0.022s sys 0.018s
I20250116 02:59:19.300490 22346 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:19.335763 22594 consensus_queue.cc:237] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:19.301900 22758 consensus_queue.cc:237] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:19.912019 20604 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.051s sys 0.071s
I20250116 02:59:19.576699 20757 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:20.010679 20604 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.077s user 0.056s sys 0.088s
W20250116 02:59:19.758560 21301 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.851s user 0.020s sys 0.025s
W20250116 02:59:20.011317 21301 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.125s user 0.026s sys 0.032s
W20250116 02:59:19.765071 21299 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.023s sys 0.024s
I20250116 02:59:19.464699 20602 raft_consensus.cc:2388] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:19.792691 20749 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.000s sys 0.051s
W20250116 02:59:19.975672 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 1.49862e+06us
W20250116 02:59:20.013965 20749 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.915s user 0.023s sys 0.071s
I20250116 02:59:19.494427 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:20.015108 20646 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:20.015501 21302 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:19.486120 20750 raft_consensus.cc:2363] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4.
I20250116 02:59:20.016765 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1448) took 1176 ms. Trace:
I20250116 02:59:19.979251 20681 raft_consensus.cc:612] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:19.356393 22600 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:19.364934 22530 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:20.001927 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:20.008538 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:19.461799 22601 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:19.959705 22775 log.cc:926] Time spent T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.407s user 0.001s sys 0.000s
W20250116 02:59:20.023450 21332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 470) took 1275 ms (client timeout 1670 ms). Trace:
I20250116 02:59:19.456615 22764 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:19.455996 22552 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:20.066552 20678 raft_consensus.cc:2405] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250116 02:59:19.761336 22711 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.106s user 0.007s sys 0.075s
W20250116 02:59:19.916350 22518 consensus_meta.cc:287] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.280s user 0.006s sys 0.007s
W20250116 02:59:20.068912 21332 rpcz_store.cc:269] 0116 02:59:18.748168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:19.372812 (+624644us) service_pool.cc:225] Handling call
0116 02:59:20.023363 (+650551us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.071810 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:20.070040 20647 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.008s sys 0.008s
I20250116 02:59:19.469700 22619 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:20.017158 20750 rpcz_store.cc:276] 0116 02:59:18.863696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:19.355639 (+491943us) service_pool.cc:225] Handling call
0116 02:59:20.016699 (+661060us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:20.010970 20757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 357) took 1344 ms (client timeout 1731 ms). Trace:
W20250116 02:59:20.073385 20752 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.007s sys 0.010s
I20250116 02:59:20.085227 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48834 (request call id 18) took 1593 ms. Trace:
W20250116 02:59:19.893321 22421 consensus_meta.cc:287] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.219s user 0.007s sys 0.002s
I20250116 02:59:20.024734 20680 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:19.762244 22687 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:19.487329 22467 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:20.087965 22711 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.430s user 0.010s sys 0.085s
W20250116 02:59:20.014104 21299 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.017s user 0.023s sys 0.035s
W20250116 02:59:20.027364 20608 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.007s sys 0.010s
W20250116 02:59:20.111641 20608 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.739s user 0.010s sys 0.011s
I20250116 02:59:20.072751 22764 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:20.088505 20647 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.682s user 0.008s sys 0.009s
I20250116 02:59:20.018693 21302 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:20.091507 20757 rpcz_store.cc:269] 0116 02:59:18.666492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.934392 (+267900us) service_pool.cc:225] Handling call
0116 02:59:20.010884 (+1076492us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:20.091651 20752 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.673s user 0.009s sys 0.010s
W20250116 02:59:20.114926 20607 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.007s sys 0.017s
I20250116 02:59:20.091923 20681 rpcz_store.cc:276] 0116 02:59:18.491584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.491759 (+ 175us) service_pool.cc:225] Handling call
0116 02:59:19.462213 (+970454us) spinlock_profiling.cc:243] Waited 36 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc85340 00007fc32bc70dd9 00007fc32bc7069c 00007fc325349d14
0116 02:59:20.085146 (+622933us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82787712}
I20250116 02:59:19.872205 22754 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:19.975076 22355 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:20.094391 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:20.127804 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1514) took 1026 ms. Trace:
I20250116 02:59:20.128187 20678 rpcz_store.cc:276] 0116 02:59:19.101006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.094010 (+993004us) service_pool.cc:225] Handling call
0116 02:59:20.127726 (+ 33716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.129446 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:19.582458 22753 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.883s user 0.024s sys 0.035s
W20250116 02:59:20.096474 20680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1500) took 1404 ms (client timeout 1737 ms). Trace:
W20250116 02:59:20.131141 20680 rpcz_store.cc:269] 0116 02:59:18.692464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.693888 (+ 1424us) service_pool.cc:225] Handling call
0116 02:59:20.001727 (+1307839us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:20.096405 (+ 94678us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3039616}
I20250116 02:59:20.096377 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:19.461165 22417 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:19.607502 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1588) took 1147 ms. Trace:
I20250116 02:59:20.012012 22532 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:19.551363 22328 raft_consensus.cc:695] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:19.514222 22740 raft_consensus.cc:695] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:19.912525 22766 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:20.007695 22612 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:20.099480 20720 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:20.019073 22679 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:20.070030 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:19.514784 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:20.075402 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1587) took 1625 ms. Trace:
W20250116 02:59:20.115479 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 96903.2us
W20250116 02:59:20.116436 20607 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.748s user 0.007s sys 0.017s
I20250116 02:59:20.168486 21302 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:20.168789 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1578) took 4603 ms. Trace:
I20250116 02:59:20.014931 20754 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:20.091414 22552 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:19.746210 22604 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:20.124150 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:20.124293 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:20.095999 22619 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:20.124317 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1848) took 1767 ms. Trace:
I20250116 02:59:20.125043 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:20.023882 22752 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:19.745899 22546 consensus_queue.cc:237] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:20.141338 20649 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.002s sys 0.025s
W20250116 02:59:20.193255 20649 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.828s user 0.002s sys 0.026s
W20250116 02:59:20.193650 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 828039us
I20250116 02:59:20.104508 22518 raft_consensus.cc:2798] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250116 02:59:20.153623 20722 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.000s sys 0.009s
I20250116 02:59:20.154630 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:20.133977 22753 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.455s user 0.030s sys 0.042s
I20250116 02:59:20.161569 20683 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 1. Current term is 2. Ops: []
W20250116 02:59:20.162518 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 63123.1us
I20250116 02:59:20.163002 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:20.164469 20751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1876) took 1542 ms (client timeout 1945 ms). Trace:
I20250116 02:59:20.179014 20752 rpcz_store.cc:276] 0116 02:59:18.357046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.359007 (+ 1961us) service_pool.cc:225] Handling call
0116 02:59:18.423150 (+ 64143us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:18.453184 (+ 30034us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:18.453223 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:18.465827 (+ 12604us) log.cc:844] Serialized 26 byte log entry
0116 02:59:18.604008 (+138181us) spinlock_profiling.cc:243] Waited 8.2 ms on lock 0x7b4800000e58. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32bbd35a1 00007fc32bbca687 00007fc32bbca51a 00007fc32bcbe9df 00007fc32bcbe6a7 00007fc32bcd2bac 00007fc32bcd2b58 00007fc32bcd2ad8 00007fc32bcd2a93 00007fc32bcd1d3c 00007fc32ac3ed16
0116 02:59:19.363839 (+759831us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:20.114844 (+751005us) log.cc:844] Serialized 14 byte log entry
0116 02:59:20.115085 (+ 241us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.115181 (+ 96us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:20.115225 (+ 44us) raft_consensus.cc:1680] finished
0116 02:59:20.124073 (+ 8848us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.124249 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35508,"prepare.queue_time_us":602935,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":549,"raft.run_cpu_time_us":1236,"raft.run_wall_time_us":144445,"spinlock_wait_cycles":18870144,"thread_start_us":1527218,"threads_started":2,"wal-append.queue_time_us":1332737}
W20250116 02:59:20.198079 20722 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.777s user 0.001s sys 0.009s
I20250116 02:59:20.205775 20720 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:20.130461 22355 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:20.169400 20607 rpcz_store.cc:276] 0116 02:59:15.565560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.325258 (+2759698us) service_pool.cc:225] Handling call
0116 02:59:18.325617 (+ 359us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:18.325998 (+ 381us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:18.326026 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:18.326549 (+ 523us) log.cc:844] Serialized 26 byte log entry
0116 02:59:19.301645 (+975096us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:20.167872 (+866227us) log.cc:844] Serialized 14 byte log entry
0116 02:59:20.168312 (+ 440us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.168423 (+ 111us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:20.168473 (+ 50us) raft_consensus.cc:1680] finished
0116 02:59:20.168485 (+ 12us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.168667 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2803,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":179,"spinlock_wait_cycles":7168,"thread_start_us":1295757,"threads_started":2,"wal-append.queue_time_us":1488495}
I20250116 02:59:20.213761 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1920) took 1598 ms. Trace:
I20250116 02:59:20.104660 22421 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.108475 22687 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:20.173578 20605 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.009s sys 0.025s
W20250116 02:59:20.217767 20605 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s user 0.010s sys 0.025s
I20250116 02:59:20.174346 20757 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:20.172133 22715 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:20.174343 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.840s user 0.008s sys 0.034s
W20250116 02:59:20.173993 20723 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.009s sys 0.026s
I20250116 02:59:20.126716 22467 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.158066 22774 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:20.235745 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 488972us
I20250116 02:59:20.136718 20600 rpcz_store.cc:276] 0116 02:59:18.446204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.747139 (+300935us) service_pool.cc:225] Handling call
0116 02:59:19.593235 (+846096us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.167412 20602 rpcz_store.cc:276] 0116 02:59:18.445972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.684836 (+238864us) service_pool.cc:225] Handling call
0116 02:59:20.070999 (+1386163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.214190 20607 rpcz_store.cc:276] 0116 02:59:18.615248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.212861 (+1597613us) service_pool.cc:225] Handling call
0116 02:59:20.213243 (+ 382us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:20.213435 (+ 192us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:20.213458 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:20.213500 (+ 42us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:20.213531 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.213584 (+ 53us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.213699 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.275766 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:20.218760 22604 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:20.218851 20757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1859) took 1621 ms (client timeout 2024 ms). Trace:
W20250116 02:59:20.276674 20757 rpcz_store.cc:269] 0116 02:59:18.597116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.123825 (+1526709us) service_pool.cc:225] Handling call
0116 02:59:20.218782 (+ 94957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.277654 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:20.174430 21332 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:20.279510 21332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 466) took 1561 ms (client timeout 1767 ms). Trace:
I20250116 02:59:20.216336 20720 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:20.157192 22665 consensus_queue.cc:1046] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20250116 02:59:20.159384 22766 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:20.224831 20723 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.818s user 0.009s sys 0.026s
W20250116 02:59:20.224519 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.890s user 0.008s sys 0.034s
I20250116 02:59:20.228343 20680 raft_consensus.cc:2405] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250116 02:59:20.129235 22754 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.232132 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:20.232781 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:20.285521 20752 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
W20250116 02:59:20.286042 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1877) took 1664 ms (client timeout 2024 ms). Trace:
W20250116 02:59:20.286387 20752 rpcz_store.cc:269] 0116 02:59:18.622016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.219207 (+1597191us) service_pool.cc:225] Handling call
0116 02:59:20.285982 (+ 66775us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:20.178750 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.024s sys 0.027s
W20250116 02:59:20.287294 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.986s user 0.024s sys 0.028s
I20250116 02:59:20.288095 21300 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:20.289199 21300 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:20.289932 21300 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:20.182749 20754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1443) took 1340 ms (client timeout 1641 ms). Trace:
I20250116 02:59:20.197961 22518 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:20.174018 22552 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:20.222117 21330 raft_consensus.cc:2388] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:20.164011 22679 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:20.214395 20751 rpcz_store.cc:269] 0116 02:59:18.621756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:19.556911 (+935155us) service_pool.cc:225] Handling call
0116 02:59:20.164396 (+607485us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:20.279865 21332 rpcz_store.cc:269] 0116 02:59:18.718323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.099087 (+1380764us) service_pool.cc:225] Handling call
0116 02:59:20.279446 (+180359us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.278321 20757 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:20.319523 20757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1873) took 1705 ms (client timeout 2122 ms). Trace:
W20250116 02:59:20.291172 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 74908.3us
I20250116 02:59:20.317209 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:20.223268 22782 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250116 02:59:20.283159 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 950200us
I20250116 02:59:20.284970 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1513) took 1203 ms. Trace:
I20250116 02:59:20.267355 21327 raft_consensus.cc:1235] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:20.301870 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1852) took 1938 ms. Trace:
I20250116 02:59:20.302632 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:20.336499 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 222925us
W20250116 02:59:20.337160 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 358) took 1670 ms (client timeout 2078 ms). Trace:
W20250116 02:59:20.337535 20752 rpcz_store.cc:269] 0116 02:59:18.666806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.287108 (+1620302us) service_pool.cc:225] Handling call
0116 02:59:20.337089 (+ 49981us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.278184 22346 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:20.310413 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 293827us
I20250116 02:59:20.284121 21328 raft_consensus.cc:1235] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:20.307255 20607 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:20.280058 22662 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
W20250116 02:59:20.319908 20757 rpcz_store.cc:269] 0116 02:59:18.614108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.277405 (+1663297us) service_pool.cc:225] Handling call
0116 02:59:20.319453 (+ 42048us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.283845 22793 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:20.332405 20681 rpcz_store.cc:276] 0116 02:59:19.081443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.211514 (+1130071us) service_pool.cc:225] Handling call
0116 02:59:20.284901 (+ 73387us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:20.335551 20756 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.427s user 0.006s sys 0.035s
I20250116 02:59:20.336902 20723 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:20.301585 22779 log.cc:926] Time spent T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.294s user 0.001s sys 0.001s
I20250116 02:59:20.236383 22715 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.340346 20755 rpcz_store.cc:276] 0116 02:59:18.363117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.363251 (+ 134us) service_pool.cc:225] Handling call
0116 02:59:18.363588 (+ 337us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:18.452229 (+ 88641us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:18.452265 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:18.452730 (+ 465us) log.cc:844] Serialized 26 byte log entry
0116 02:59:18.603420 (+150690us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b4800000e58. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32bbd35a1 00007fc32bbca687 00007fc32bbca51a 00007fc32bcbe9df 00007fc32bcbe6a7 00007fc32bcd2bac 00007fc32bcd2b58 00007fc32bcd2ad8 00007fc32bcd2a93 00007fc32bcd1d3c 00007fc32ac3ed16
0116 02:59:19.976409 (+1372989us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:20.069369 (+ 92960us) log.cc:844] Serialized 14 byte log entry
0116 02:59:20.295020 (+225651us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.295123 (+ 103us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:20.295165 (+ 42us) raft_consensus.cc:1680] finished
0116 02:59:20.301614 (+ 6449us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.301791 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19905,"prepare.queue_time_us":128600,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":131752,"raft.run_cpu_time_us":1266,"raft.run_wall_time_us":157502,"spinlock_wait_cycles":68551552,"thread_start_us":1735038,"threads_started":2,"wal-append.queue_time_us":1430552}
I20250116 02:59:20.301276 22518 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:20.301402 22774 raft_consensus.cc:2798] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:20.301119 22753 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:20.310135 22748 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:20.299072 22792 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:20.368976 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:20.308391 22552 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:20.372726 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:20.375507 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:20.232414 22328 consensus_queue.cc:237] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.376950 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:20.375312 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:20.331550 22782 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:20.295195 20754 rpcz_store.cc:269] 0116 02:59:18.839013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:19.155556 (+316543us) service_pool.cc:225] Handling call
0116 02:59:20.179213 (+1023657us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:20.463136 20751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1439) took 1784 ms (client timeout 2098 ms). Trace:
W20250116 02:59:20.463526 20751 rpcz_store.cc:269] 0116 02:59:18.678631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.331280 (+1652649us) service_pool.cc:225] Handling call
0116 02:59:20.463057 (+131777us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:20.373934 20756 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.465s user 0.007s sys 0.035s
I20250116 02:59:20.402354 21300 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:20.276786 22740 consensus_queue.cc:237] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:20.411367 20681 raft_consensus.cc:1270] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250116 02:59:20.434923 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:20.468658 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1511) took 1536 ms. Trace:
I20250116 02:59:20.339198 22796 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:20.342450 21330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 465) took 1624 ms (client timeout 1737 ms). Trace:
I20250116 02:59:20.372583 22600 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:20.442121 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 121068us
I20250116 02:59:20.449841 20752 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:20.415922 22753 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:20.459157 20757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 356) took 1793 ms (client timeout 2183 ms). Trace:
I20250116 02:59:20.482631 20720 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:20.465121 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 62866.7us
I20250116 02:59:20.421911 22774 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:20.442281 22749 consensus_meta.cc:287] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.782s user 0.014s sys 0.000s
I20250116 02:59:20.469033 20681 rpcz_store.cc:276] 0116 02:59:18.931802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.373290 (+1441488us) service_pool.cc:225] Handling call
0116 02:59:20.410918 (+ 37628us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:20.468325 (+ 57407us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.468591 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.468950 20755 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:20.445675 22667 consensus_meta.cc:287] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.592s user 0.014s sys 0.000s
W20250116 02:59:20.458163 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 121330us
I20250116 02:59:20.493846 20723 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 02:59:20.476527 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1450) took 1524 ms (client timeout 1885 ms). Trace:
I20250116 02:59:20.481712 20680 raft_consensus.cc:1270] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:59:20.486656 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:20.482946 20757 rpcz_store.cc:269] 0116 02:59:18.666154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.375014 (+1708860us) service_pool.cc:225] Handling call
0116 02:59:20.459076 (+ 84062us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:20.491370 20755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1449) took 1554 ms (client timeout 1972 ms). Trace:
I20250116 02:59:20.489941 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1853) took 2043 ms. Trace:
I20250116 02:59:20.460899 22748 raft_consensus.cc:695] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:20.510403 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1518) took 1257 ms. Trace:
I20250116 02:59:20.461437 22417 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:20.468040 22784 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:20.395648 22793 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.462267 22782 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:20.804656 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 274385us
W20250116 02:59:20.495209 22783 log.cc:926] Time spent T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.365s user 0.001s sys 0.000s
W20250116 02:59:20.790822 22728 consensus_meta.cc:287] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.936s user 0.020s sys 0.002s
I20250116 02:59:20.803488 22421 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:20.656221 22807 log.cc:926] Time spent T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20250116 02:59:20.599674 22804 log.cc:926] Time spent T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8: Append to log took a long time: real 0.070s user 0.000s sys 0.002s
W20250116 02:59:20.785223 22619 consensus_meta.cc:287] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.586s user 0.009s sys 0.001s
I20250116 02:59:20.736840 22824 raft_consensus.cc:2798] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:20.487278 22601 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:20.488782 22774 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:20.485813 22753 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:20.737283 22815 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:20.601610 22754 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:20.830453 22696 consensus_meta.cc:287] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.800s user 0.006s sys 0.006s
I20250116 02:59:20.527747 22790 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:20.741497 22679 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:20.600842 22766 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:20.488417 22600 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:20.586158 22687 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:20.602164 22823 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250116 02:59:20.842533 22660 consensus_meta.cc:287] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.819s user 0.004s sys 0.008s
I20250116 02:59:20.586889 22791 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:20.568151 22775 log.cc:926] Time spent T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
W20250116 02:59:20.527936 22781 log.cc:926] Time spent T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.221s user 0.000s sys 0.000s
I20250116 02:59:20.849309 22793 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:20.850769 22755 consensus_meta.cc:287] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.782s user 0.011s sys 0.000s
W20250116 02:59:20.851727 22808 log.cc:926] Time spent T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.323s user 0.003s sys 0.000s
I20250116 02:59:20.510897 22749 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.568930 22752 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:20.402304 20572 leader_election.cc:336] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.641s (SENT)
I20250116 02:59:20.339461 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:21.003387 22755 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.999969 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34864 (request call id 1159) took 2322 ms. Trace:
I20250116 02:59:21.031458 20678 raft_consensus.cc:2388] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:20.790467 22552 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:21.035847 20600 raft_consensus.cc:2405] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20250116 02:59:21.037530 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:21.030402 20679 rpcz_store.cc:276] 0116 02:59:18.677787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.695567 (+ 17780us) service_pool.cc:225] Handling call
0116 02:59:20.946902 (+2251335us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe7a6 00007fc32bcbe980 00007fc32bc71b60 00007fc32534e817 00007fc32534e7a2 00007fc32534e6d1
0116 02:59:20.999651 (+ 52749us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86481280}
I20250116 02:59:21.041600 22755 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:20.932790 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:20.934290 20603 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.040s user 0.000s sys 0.004s
I20250116 02:59:20.935593 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1911) took 2741 ms. Trace:
I20250116 02:59:20.936702 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1916) took 2709 ms. Trace:
I20250116 02:59:20.941114 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1586) took 2495 ms. Trace:
W20250116 02:59:20.943557 20646 leader_election.cc:336] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Timed out: RequestConsensusVote RPC to 127.18.238.132:33959 timed out after 1.576s (SENT)
W20250116 02:59:20.944625 20647 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.012s sys 0.010s
I20250116 02:59:20.511253 20680 rpcz_store.cc:276] 0116 02:59:19.253037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.449261 (+1196224us) service_pool.cc:225] Handling call
0116 02:59:20.481353 (+ 32092us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:20.510047 (+ 28694us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.510326 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.946146 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:21.049203 20675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1507) took 2302 ms (client timeout 1770 ms). Trace:
W20250116 02:59:21.049613 20675 rpcz_store.cc:269] 0116 02:59:18.746955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.851942 (+104987us) service_pool.cc:225] Handling call
0116 02:59:20.484668 (+1632726us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:21.049053 (+564385us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87100800}
I20250116 02:59:20.505915 20677 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:21.047590 20608 rpcz_store.cc:276] 0116 02:59:18.445731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:19.126585 (+680854us) service_pool.cc:225] Handling call
0116 02:59:19.372397 (+245812us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:20.112169 (+739772us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:20.112205 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:20.126170 (+ 13965us) log.cc:844] Serialized 26 byte log entry
0116 02:59:20.337927 (+211757us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:20.337968 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.338044 (+ 76us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:20.851108 (+513064us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:20.940354 (+ 89246us) raft_consensus.cc:1680] finished
0116 02:59:20.940395 (+ 41us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.940898 (+ 503us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3037,"raft.queue_time_us":97275,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":16644,"spinlock_wait_cycles":2465152,"thread_start_us":889787,"threads_started":2,"wal-append.queue_time_us":394830}
I20250116 02:59:21.052521 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:21.052884 20677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1512) took 2100 ms (client timeout 1700 ms). Trace:
I20250116 02:59:20.949507 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1502) took 2256 ms. Trace:
W20250116 02:59:21.054322 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 651842us
I20250116 02:59:21.031306 22810 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:21.053936 20683 rpcz_store.cc:276] 0116 02:59:18.693040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.693162 (+ 122us) service_pool.cc:225] Handling call
0116 02:59:19.269839 (+576677us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:19.270129 (+ 290us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:20.208035 (+937906us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.948614 (+740579us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b5400086bd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:20.949145 (+ 531us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2127782528}
W20250116 02:59:20.945186 20649 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.002s sys 0.011s
W20250116 02:59:21.056195 20649 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.669s user 0.003s sys 0.011s
I20250116 02:59:21.055831 22810 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:21.056640 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 669777us
W20250116 02:59:21.053319 20603 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.159s user 0.001s sys 0.004s
I20250116 02:59:20.944229 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:21.059616 20607 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:21.061146 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:21.053715 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 110548us
I20250116 02:59:21.062181 20607 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 4
I20250116 02:59:20.527155 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1857) took 1930 ms. Trace:
I20250116 02:59:21.047266 20605 rpcz_store.cc:276] 0116 02:59:18.227651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.938528 (+710877us) service_pool.cc:225] Handling call
0116 02:59:18.938899 (+ 371us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:19.314903 (+376004us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:19.314938 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:19.405555 (+ 90617us) log.cc:844] Serialized 26 byte log entry
0116 02:59:20.218311 (+812756us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:20.218354 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.218427 (+ 73us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:20.936175 (+717748us) raft_consensus.cc:1680] finished
0116 02:59:20.936209 (+ 34us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.936506 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":394804,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":27658,"thread_start_us":723714,"threads_started":1,"wal-append.queue_time_us":1387998}
I20250116 02:59:21.064687 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:21.063467 20748 rpcz_store.cc:276] 0116 02:59:18.596501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.598715 (+ 2214us) service_pool.cc:225] Handling call
0116 02:59:18.599055 (+ 340us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:18.599294 (+ 239us) raft_consensus.cc:1529] Early marking committed up to index 4
0116 02:59:20.024419 (+1425125us) log.cc:844] Serialized 14 byte log entry
0116 02:59:20.467750 (+443331us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:20.478943 (+ 11193us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:20.484502 (+ 5559us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.491817 (+ 7315us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.527069 (+ 35252us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1221792,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":167965,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":6054,"spinlock_wait_cycles":37944832,"thread_start_us":1746009,"threads_started":3,"wal-append.queue_time_us":344983}
I20250116 02:59:21.065711 20605 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:20.953275 20752 rpcz_store.cc:269] 0116 02:59:18.952486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.366411 (+1413925us) service_pool.cc:225] Handling call
0116 02:59:20.476453 (+110042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:21.067036 20602 raft_consensus.cc:2371] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4. Re-sending same reply.
W20250116 02:59:20.954057 20751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 361) took 2274 ms (client timeout 2231 ms). Trace:
I20250116 02:59:20.531056 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:21.037323 22518 consensus_meta.cc:287] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.617s user 0.010s sys 0.014s
W20250116 02:59:20.560556 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 66785.1us
I20250116 02:59:20.740454 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 701) took 2048 ms. Trace:
I20250116 02:59:20.961149 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 700) took 2283 ms. Trace:
W20250116 02:59:20.954248 20755 rpcz_store.cc:269] 0116 02:59:18.936650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.375790 (+1439140us) service_pool.cc:225] Handling call
0116 02:59:20.491303 (+115513us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:20.476804 21330 rpcz_store.cc:269] 0116 02:59:18.718074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:19.377886 (+659812us) service_pool.cc:225] Handling call
0116 02:59:20.337233 (+959347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.813432 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:20.954625 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1851) took 2596 ms. Trace:
I20250116 02:59:21.061532 21302 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:20.963840 21332 raft_consensus.cc:2388] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:20.509649 20756 rpcz_store.cc:276] 0116 02:59:18.446545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.447652 (+ 1107us) service_pool.cc:225] Handling call
0116 02:59:18.447998 (+ 346us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:18.448380 (+ 382us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:18.448409 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:18.462574 (+ 14165us) log.cc:844] Serialized 26 byte log entry
0116 02:59:18.602682 (+140108us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b4800000e58. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32bbd35a1 00007fc32bbca687 00007fc32bbca51a 00007fc32bcbe9df 00007fc32bcbe6a7 00007fc32bcd2bac 00007fc32bcd2b58 00007fc32bcd2ad8 00007fc32bcd2a93 00007fc32bcd1d3c 00007fc32ac3ed16
0116 02:59:18.728791 (+126109us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:20.464629 (+1735838us) log.cc:844] Serialized 14 byte log entry
0116 02:59:20.474275 (+ 9646us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.474360 (+ 85us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:20.474405 (+ 45us) raft_consensus.cc:1680] finished
0116 02:59:20.489695 (+ 15290us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.489877 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9375,"prepare.queue_time_us":1142661,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":271739,"raft.run_cpu_time_us":2761,"raft.run_wall_time_us":149420,"spinlock_wait_cycles":7230208,"thread_start_us":1766551,"threads_started":2,"wal-append.queue_time_us":947390}
W20250116 02:59:20.822891 22530 consensus_meta.cc:287] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.727s user 0.012s sys 0.000s
W20250116 02:59:20.828354 22622 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.357s user 0.000s sys 0.026s
W20250116 02:59:21.100095 22622 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.634s user 0.000s sys 0.027s
I20250116 02:59:20.834903 22601 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:21.101733 22601 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:20.851305 22696 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.510668 22665 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:20.510159 22667 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:21.107466 21335 raft_consensus.cc:1235] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 4. Ops: []
I20250116 02:59:21.108717 22667 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:20.852344 22715 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:20.813686 22728 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:20.974084 22467 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:20.860073 22600 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:20.978894 22660 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.980824 22599 raft_consensus.cc:988] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:20.838450 22619 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:21.139220 22599 raft_consensus.cc:1076] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:21.140471 20607 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:21.141045 20756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1447) took 2300 ms (client timeout 2225 ms). Trace:
I20250116 02:59:21.141237 22619 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:20.878466 22774 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.887236 22782 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:20.529165 22787 log.cc:926] Time spent T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.252s user 0.001s sys 0.000s
I20250116 02:59:20.860468 22790 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:21.143271 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1878) took 2521 ms (client timeout 2398 ms). Trace:
W20250116 02:59:21.143769 20752 rpcz_store.cc:269] 0116 02:59:18.622272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.142986 (+2520714us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.143175 (+ 189us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250116 02:59:21.144846 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1452) took 1991 ms (client timeout 1916 ms). Trace:
I20250116 02:59:20.868871 22791 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.545842 22792 consensus_queue.cc:1035] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.316s
I20250116 02:59:20.480502 22796 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.532171 22801 raft_consensus.cc:3049] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:20.841322 22815 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:20.863126 22823 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:20.858949 22824 raft_consensus.cc:695] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:20.970168 22441 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:20.741847 22748 consensus_queue.cc:237] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:20.997125 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1919) took 2386 ms. Trace:
W20250116 02:59:20.997107 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.008s sys 0.011s
I20250116 02:59:21.024367 22749 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:21.026857 22766 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:20.928512 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1910) took 2734 ms. Trace:
W20250116 02:59:20.928885 20601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 1890) took 5905 ms (client timeout 2398 ms). Trace:
I20250116 02:59:21.042237 22690 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:20.865962 22747 consensus_meta.cc:287] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.781s user 0.009s sys 0.003s
I20250116 02:59:21.046914 20604 rpcz_store.cc:276] 0116 02:59:18.193720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.910291 (+716571us) service_pool.cc:225] Handling call
0116 02:59:18.910634 (+ 343us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:18.910980 (+ 346us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:18.911009 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:18.933173 (+ 22164us) log.cc:844] Serialized 26 byte log entry
0116 02:59:20.011800 (+1078627us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:20.342000 (+330200us) log.cc:844] Serialized 14 byte log entry
0116 02:59:20.443942 (+101942us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.444019 (+ 77us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:20.934855 (+490836us) raft_consensus.cc:1680] finished
0116 02:59:20.934905 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.935318 (+ 413us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":233292,"prepare.queue_time_us":516450,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":143300,"raft.run_cpu_time_us":531,"raft.run_wall_time_us":60097,"thread_start_us":1431449,"threads_started":3,"wal-append.queue_time_us":1148452}
I20250116 02:59:21.053892 20608 raft_consensus.cc:2405] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:20.955964 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:20.002671 20674 rpcz_store.cc:276] 0116 02:59:15.231683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.233976 (+ 2293us) service_pool.cc:225] Handling call
0116 02:59:15.262708 (+ 28732us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:15.262963 (+ 255us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:17.840777 (+2577814us) log.cc:844] Serialized 14 byte log entry
0116 02:59:18.729616 (+888839us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:18.747839 (+ 18223us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:18.762178 (+ 14339us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:18.857014 (+ 94836us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:18.933651 (+ 76637us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21841,"prepare.queue_time_us":54881,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":88120,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":55332,"thread_start_us":3342513,"threads_started":2,"wal-append.queue_time_us":543105,"wal-append.run_cpu_time_us":1326,"wal-append.run_wall_time_us":1016271}
W20250116 02:59:20.561332 21299 leader_election.cc:336] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 1.694s (SENT)
W20250116 02:59:21.048368 20647 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.711s user 0.013s sys 0.010s
W20250116 02:59:21.053413 20677 rpcz_store.cc:269] 0116 02:59:18.952486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:19.331595 (+379109us) service_pool.cc:225] Handling call
0116 02:59:21.052410 (+1720815us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b54000866d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:21.052611 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1690523904}
W20250116 02:59:21.067803 20751 rpcz_store.cc:269] 0116 02:59:18.679515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.464202 (+1784687us) service_pool.cc:225] Handling call
0116 02:59:20.953691 (+489489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:20.963747 21331 raft_consensus.cc:1235] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:21.064530 22552 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:21.043814 20600 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:21.098562 22530 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:21.103291 20720 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.012s sys 0.006s
W20250116 02:59:21.077297 20748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1440) took 2231 ms (client timeout 2178 ms). Trace:
W20250116 02:59:21.119758 21330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 709) took 2386 ms (client timeout 2122 ms). Trace:
I20250116 02:59:21.077414 20753 rpcz_store.cc:276] 0116 02:59:18.357897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.361612 (+ 3715us) service_pool.cc:225] Handling call
0116 02:59:18.361975 (+ 363us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:18.420236 (+ 58261us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:18.420268 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:18.452084 (+ 31816us) log.cc:844] Serialized 26 byte log entry
0116 02:59:18.503890 (+ 51806us) spinlock_profiling.cc:243] Waited 8.03 ms on lock 0x7b180041f860. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32698be4b 00007fc326991913 00007fc326991cc1 00007fc326ce345f
0116 02:59:20.015784 (+1511894us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:20.100270 (+ 84486us) log.cc:844] Serialized 14 byte log entry
0116 02:59:20.100499 (+ 229us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.100583 (+ 84us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:20.530256 (+429673us) raft_consensus.cc:1680] finished
0116 02:59:20.551086 (+ 20830us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.871849 (+320763us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b180041f860. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323da3ec6 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd6914 00007fc323dd9cc1
0116 02:59:20.954315 (+ 82466us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115254,"prepare.queue_time_us":144207,"prepare.run_wall_time_us":3615,"raft.queue_time_us":159717,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":16176,"spinlock_wait_cycles":69784448,"thread_start_us":1473930,"threads_started":1,"wal-append.queue_time_us":1451353}
I20250116 02:59:21.077003 21328 rpcz_store.cc:276] 0116 02:59:18.677151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.702712 (+ 25561us) service_pool.cc:225] Handling call
0116 02:59:20.203052 (+1500340us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:20.225320 (+ 22268us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:20.374049 (+148729us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.960259 (+586210us) spinlock_profiling.cc:243] Waited 37.5 ms on lock 0x7b54006108d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:20.960818 (+ 559us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123565312}
I20250116 02:59:21.124258 22810 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:21.131546 21329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 704) took 2432 ms (client timeout 1666 ms). Trace:
W20250116 02:59:21.141556 20756 rpcz_store.cc:269] 0116 02:59:18.840139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.140657 (+2300518us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.140816 (+ 159us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:21.091126 22518 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:21.441077 20605 raft_consensus.cc:2405] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250116 02:59:21.145179 20752 rpcz_store.cc:269] 0116 02:59:19.153937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.144642 (+1990705us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.144779 (+ 137us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250116 02:59:21.449471 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 387071us
I20250116 02:59:21.157317 21336 raft_consensus.cc:2388] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:21.378294 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:21.309453 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 463) took 2612 ms. Trace:
I20250116 02:59:21.145207 22774 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:21.160135 20678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1523) took 1651 ms (client timeout 1898 ms). Trace:
I20250116 02:59:21.487491 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 324) took 1233 ms. Trace:
W20250116 02:59:21.477780 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.062s user 0.009s sys 0.027s
W20250116 02:59:21.300452 22633 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.777s user 0.017s sys 0.025s
I20250116 02:59:21.145991 20754 raft_consensus.cc:2388] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 2: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:21.323707 22805 log.cc:926] Time spent T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.846s user 0.005s sys 0.003s
W20250116 02:59:21.472066 20601 rpcz_store.cc:269] 0116 02:59:15.023236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:15.031659 (+ 8423us) service_pool.cc:225] Handling call
0116 02:59:20.928313 (+5896654us) spinlock_profiling.cc:243] Waited 3.63 s on lock 0x7b540024f4d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:20.928592 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8345992448}
W20250116 02:59:21.156610 20755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1453) took 1823 ms (client timeout 1789 ms). Trace:
W20250116 02:59:21.455724 22355 consensus_meta.cc:287] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.196s user 0.000s sys 0.027s
W20250116 02:59:21.488745 20720 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.005s user 0.016s sys 0.022s
W20250116 02:59:21.311786 21332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 710) took 2550 ms (client timeout 1932 ms). Trace:
W20250116 02:59:21.274351 20648 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.000s sys 0.008s
W20250116 02:59:21.158519 22604 consensus_meta.cc:287] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.873s user 0.013s sys 0.010s
I20250116 02:59:21.488001 20608 rpcz_store.cc:276] 0116 02:59:20.241711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.051959 (+810248us) service_pool.cc:225] Handling call
0116 02:59:21.474760 (+422801us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:21.475963 21335 rpcz_store.cc:276] 0116 02:59:18.697113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.697254 (+ 141us) service_pool.cc:225] Handling call
0116 02:59:18.748649 (+ 51395us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:21.108478 (+2359829us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:21.309071 (+200593us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b540078f5d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:21.309372 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5423134976}
W20250116 02:59:21.149132 22773 log.cc:926] Time spent T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 1.776s user 0.008s sys 0.008s
I20250116 02:59:21.488504 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 322) took 1233 ms. Trace:
W20250116 02:59:21.496430 20755 rpcz_store.cc:269] 0116 02:59:19.333356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.156340 (+1822984us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.156514 (+ 174us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:21.138322 22755 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:21.324721 20572 leader_election.cc:336] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.770s (SENT)
I20250116 02:59:21.454244 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:21.471895 20606 rpcz_store.cc:276] 0116 02:59:18.611041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:19.339584 (+728543us) service_pool.cc:225] Handling call
0116 02:59:19.354949 (+ 15365us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:19.358739 (+ 3790us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:19.358771 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:20.937126 (+1578355us) spinlock_profiling.cc:243] Waited 49.4 ms on lock 0x7b2c000278a8. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc3256f0741 00007fc3256ffd72 00007fc326d4b141 00007fc326d0f58a 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782
0116 02:59:20.939347 (+ 2221us) log.cc:844] Serialized 572 byte log entry
0116 02:59:20.993277 (+ 53930us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:20.993340 (+ 63us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.993453 (+ 113us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:20.996434 (+ 2981us) raft_consensus.cc:1680] finished
0116 02:59:20.996457 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.996905 (+ 448us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113565568,"thread_start_us":53314,"threads_started":1,"wal-append.queue_time_us":53880}
I20250116 02:59:21.496750 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 317) took 3048 ms. Trace:
I20250116 02:59:21.277148 22667 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:21.518929 20605 rpcz_store.cc:276] 0116 02:59:20.241403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.064275 (+822872us) service_pool.cc:225] Handling call
0116 02:59:21.475101 (+410826us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:21.517525 20756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1458) took 1895 ms (client timeout 1641 ms). Trace:
W20250116 02:59:21.517333 20753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1874) took 2902 ms (client timeout 2636 ms). Trace:
W20250116 02:59:21.522722 20756 rpcz_store.cc:269] 0116 02:59:19.598402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.489624 (+1891222us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.493961 (+ 4337us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250116 02:59:21.492985 22633 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 2.180s user 0.021s sys 0.029s
W20250116 02:59:21.491206 20678 rpcz_store.cc:269] 0116 02:59:19.509265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.129141 (+619876us) service_pool.cc:225] Handling call
0116 02:59:21.159797 (+1030656us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b5400325354. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:21.160028 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2071228800}
W20250116 02:59:21.522238 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 197612us
W20250116 02:59:21.522905 20753 rpcz_store.cc:269] 0116 02:59:18.614388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.493859 (+2879471us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.517214 (+ 23355us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250116 02:59:21.496249 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 435052us
I20250116 02:59:21.157209 22665 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:21.525029 20607 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:21.525363 22801 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:21.525985 21336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 468) took 2797 ms (client timeout 1596 ms). Trace:
W20250116 02:59:21.526526 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 1.0439e+06us
I20250116 02:59:21.526753 22815 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:21.145373 22790 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:21.495602 22766 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:21.524231 20648 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.064s user 0.009s sys 0.022s
I20250116 02:59:21.455462 22696 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:21.497006 21332 rpcz_store.cc:269] 0116 02:59:18.773286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.336079 (+1562793us) service_pool.cc:225] Handling call
0116 02:59:21.311433 (+975354us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540058cd54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:21.311625 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1186456448}
I20250116 02:59:21.488888 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 1925) took 1100 ms. Trace:
W20250116 02:59:21.521646 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1446) took 2681 ms (client timeout 2534 ms). Trace:
W20250116 02:59:22.107573 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.014s sys 0.030s
I20250116 02:59:21.520584 20603 rpcz_store.cc:276] 0116 02:59:18.444257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:19.101350 (+657093us) service_pool.cc:225] Handling call
0116 02:59:19.153685 (+ 52335us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:19.357433 (+203748us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:19.357569 (+ 136us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:19.358430 (+ 861us) log.cc:844] Serialized 26 byte log entry
0116 02:59:19.795470 (+437040us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:21.057634 (+1262164us) log.cc:844] Serialized 14 byte log entry
0116 02:59:21.058205 (+ 571us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:21.058351 (+ 146us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:21.145561 (+ 87210us) raft_consensus.cc:1680] finished
0116 02:59:21.145581 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:21.492826 (+347245us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":203013,"prepare.queue_time_us":732476,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":56,"raft.queue_time_us":224621,"raft.run_cpu_time_us":520,"raft.run_wall_time_us":49619,"thread_start_us":1532646,"threads_started":2,"wal-append.queue_time_us":828968}
W20250116 02:59:21.644773 20756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1462) took 1633 ms (client timeout 1516 ms). Trace:
W20250116 02:59:21.611295 20751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1459) took 1897 ms (client timeout 1770 ms). Trace:
I20250116 02:59:21.126756 21327 rpcz_store.cc:276] 0116 02:59:18.691671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.702596 (+ 10925us) service_pool.cc:225] Handling call
0116 02:59:20.023348 (+1320752us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:20.091934 (+ 68586us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:20.344037 (+252103us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.740060 (+396023us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54006f13d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:20.740360 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":325107840}
W20250116 02:59:21.472690 21330 rpcz_store.cc:269] 0116 02:59:18.733592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.119299 (+2385707us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.119484 (+ 185us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:21.530453 20676 raft_consensus.cc:2363] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4.
W20250116 02:59:21.851857 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 83830.2us
I20250116 02:59:21.624063 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:22.110869 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 506876us
W20250116 02:59:22.121518 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.017s sys 0.030s
W20250116 02:59:21.914743 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 1.16702e+06us
W20250116 02:59:22.130539 20753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1456) took 2635 ms (client timeout 2097 ms). Trace:
I20250116 02:59:21.746743 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1849) took 3279 ms. Trace:
W20250116 02:59:21.647260 21335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 467) took 2917 ms (client timeout 2191 ms). Trace:
W20250116 02:59:22.126406 20756 rpcz_store.cc:269] 0116 02:59:20.008210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.624415 (+1616205us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.641071 (+ 16656us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250116 02:59:22.138054 20753 rpcz_store.cc:269] 0116 02:59:19.495042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.093102 (+2598060us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:22.130467 (+ 37365us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:21.496024 22660 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:21.757570 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 80051us
W20250116 02:59:21.472224 20748 rpcz_store.cc:269] 0116 02:59:18.837945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.066344 (+2228399us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.066595 (+ 251us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250116 02:59:21.764827 21333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 702) took 3072 ms (client timeout 1576 ms). Trace:
W20250116 02:59:21.622529 20755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1869) took 3009 ms (client timeout 2888 ms). Trace:
W20250116 02:59:22.124641 20752 rpcz_store.cc:269] 0116 02:59:18.839871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.521404 (+2681533us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.521574 (+ 170us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:21.793673 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1921) took 1574 ms. Trace:
W20250116 02:59:22.121515 21336 rpcz_store.cc:269] 0116 02:59:18.728216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.915342 (+187126us) service_pool.cc:225] Handling call
0116 02:59:21.525649 (+2610307us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b54005c3ad4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:21.525877 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5009387392}
W20250116 02:59:22.126185 20751 rpcz_store.cc:269] 0116 02:59:19.694358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.527757 (+1833399us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.591610 (+ 63853us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250116 02:59:21.646423 20754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1455) took 2159 ms (client timeout 1513 ms). Trace:
W20250116 02:59:21.839581 22838 log.cc:926] Time spent T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.347s user 0.002s sys 0.000s
W20250116 02:59:21.529922 20682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1508) took 2755 ms (client timeout 1641 ms). Trace:
I20250116 02:59:21.535025 22825 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:22.089011 21302 leader_election.cc:336] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.746s (SENT)
I20250116 02:59:21.529309 22823 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:21.888440 22842 log.cc:926] Time spent T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.093s user 0.000s sys 0.002s
W20250116 02:59:22.182024 21335 rpcz_store.cc:269] 0116 02:59:18.727658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.619155 (+2891497us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.645328 (+ 26173us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250116 02:59:22.141703 20756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1457) took 2582 ms (client timeout 2059 ms). Trace:
W20250116 02:59:22.180954 20755 rpcz_store.cc:269] 0116 02:59:18.613058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.610750 (+2997692us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.622432 (+ 11682us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250116 02:59:21.537968 22753 consensus_meta.cc:287] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.682s user 0.005s sys 0.008s
I20250116 02:59:22.144620 20503 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48594 (request call id 217) took 10386 ms. Trace:
I20250116 02:59:22.181778 20604 rpcz_store.cc:276] 0116 02:59:20.195040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.640432 (+1445392us) service_pool.cc:225] Handling call
0116 02:59:21.678099 (+ 37667us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:21.750588 (+ 72489us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:21.750644 (+ 56us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:21.750702 (+ 58us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:21.769069 (+ 18367us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:21.769158 (+ 89us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:21.769296 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:22.159766 20753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1879) took 3516 ms (client timeout 3182 ms). Trace:
W20250116 02:59:22.196764 20753 rpcz_store.cc:269] 0116 02:59:18.643653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.139324 (+3495671us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:22.159677 (+ 20353us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:22.172504 20502 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41978 (request call id 325) took 8800 ms. Trace:
W20250116 02:59:21.840880 22612 consensus_meta.cc:287] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.667s user 0.017s sys 0.007s
W20250116 02:59:22.194973 20756 rpcz_store.cc:269] 0116 02:59:19.558956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.141474 (+2582518us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:22.141627 (+ 153us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:22.194926 20607 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:22.090570 22796 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:22.151613 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1922) took 1954 ms. Trace:
W20250116 02:59:22.146705 22764 consensus_meta.cc:287] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.723s user 0.013s sys 0.018s
I20250116 02:59:21.539861 22782 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:22.165767 22601 consensus_meta.cc:287] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.063s user 0.007s sys 0.006s
I20250116 02:59:22.087513 22749 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:21.766049 22530 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:22.193871 20682 rpcz_store.cc:269] 0116 02:59:18.774321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.851689 (+ 77368us) service_pool.cc:225] Handling call
0116 02:59:19.270636 (+418947us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:21.529441 (+2258805us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b54001bbf54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:21.529676 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4111072896}
I20250116 02:59:21.471350 20599 rpcz_store.cc:276] 0116 02:59:18.193479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.366494 (+2173015us) service_pool.cc:225] Handling call
0116 02:59:20.438567 (+ 72073us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:20.927299 (+488732us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:20.927401 (+ 102us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:20.927508 (+ 107us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:20.927581 (+ 73us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.927749 (+ 168us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:20.928141 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.140715 20571 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:21.543895 22690 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:22.181147 21333 rpcz_store.cc:269] 0116 02:59:18.692175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.692973 (+ 798us) service_pool.cc:225] Handling call
0116 02:59:21.734586 (+3041613us) spinlock_profiling.cc:243] Waited 2.62 s on lock 0x7b54006900d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:21.764577 (+ 29991us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6015684992}
W20250116 02:59:22.170583 21332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 475) took 2870 ms (client timeout 1898 ms). Trace:
W20250116 02:59:21.664402 21328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 474) took 2364 ms (client timeout 1718 ms). Trace:
W20250116 02:59:21.472482 21329 rpcz_store.cc:269] 0116 02:59:18.694391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.729310 (+ 34919us) service_pool.cc:225] Handling call
0116 02:59:20.811083 (+2081773us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:21.126981 (+315898us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36518656}
I20250116 02:59:21.535645 22791 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:22.216357 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:21.766912 21331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 699) took 3090 ms. Trace:
I20250116 02:59:22.201937 20605 rpcz_store.cc:276] 0116 02:59:20.197087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.150332 (+1953245us) service_pool.cc:225] Handling call
0116 02:59:22.150849 (+ 517us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.151206 (+ 357us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.151232 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.151280 (+ 48us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:22.151326 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.151386 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.151518 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.202570 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:21.531486 22600 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:22.214532 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 73969.4us
W20250116 02:59:22.145826 20676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1522) took 2662 ms (client timeout 1718 ms). Trace:
I20250116 02:59:21.545620 20602 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:21.641886 22728 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:21.541805 22518 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:22.202921 20751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1460) took 2380 ms (client timeout 2098 ms). Trace:
W20250116 02:59:22.228137 20751 rpcz_store.cc:269] 0116 02:59:19.816850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.189313 (+2372463us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:22.196935 (+ 7622us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:22.226760 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:21.604209 22604 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:22.171499 21330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 472) took 3296 ms (client timeout 2328 ms). Trace:
I20250116 02:59:22.197742 20502 rpcz_store.cc:276] 0116 02:59:13.371721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:13.372559 (+ 838us) service_pool.cc:225] Handling call
0116 02:59:15.408904 (+2036345us) spinlock_profiling.cc:243] Waited 99.5 ms on lock 0x7b54000af5d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c94d50 00007fc326d17f2a 00007fc32ca33dec 00007fc32cb2bdee 00007fc32b07efc4 00007fc32b07ef52 00007fc32b07ee81 00007fc32b07edfc 00007fc32b07e0b2 00007fc32513adfc 00007fc32513a236
0116 02:59:22.172235 (+6763331us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69029,"reported_tablets":9,"spinlock_wait_cycles":228854528}
I20250116 02:59:22.212474 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:22.212826 21332 rpcz_store.cc:269] 0116 02:59:19.300161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.170323 (+2870162us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:22.170500 (+ 177us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:21.679076 22755 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:22.211000 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:21.914376 22800 consensus_queue.cc:1046] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.294s
I20250116 02:59:21.536710 22715 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:22.200450 22790 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:22.221848 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:22.223806 20571 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:22.216974 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.124331 20600 rpcz_store.cc:276] 0116 02:59:20.386779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.037018 (+650239us) service_pool.cc:225] Handling call
0116 02:59:21.477410 (+440392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.221638 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:21.528198 22355 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:21.840430 22747 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:22.189656 20754 rpcz_store.cc:269] 0116 02:59:19.481613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.481308 (+999695us) service_pool.cc:225] Handling call
0116 02:59:21.628170 (+1146862us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.204959 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 1930) took 1065 ms. Trace:
W20250116 02:59:22.219478 20676 rpcz_store.cc:269] 0116 02:59:19.483440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.058303 (+574863us) service_pool.cc:225] Handling call
0116 02:59:22.145559 (+2087256us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54002fdd54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:22.145750 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3130731904}
I20250116 02:59:22.236117 20604 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 5
I20250116 02:59:22.227206 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 1926) took 1840 ms. Trace:
W20250116 02:59:22.215308 21328 rpcz_store.cc:269] 0116 02:59:19.299853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.664066 (+2364213us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:21.664276 (+ 210us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:22.180406 20749 rpcz_store.cc:276] 0116 02:59:18.357355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.359116 (+ 1761us) service_pool.cc:225] Handling call
0116 02:59:18.359487 (+ 371us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:18.407476 (+ 47989us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:18.407512 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:18.408038 (+ 526us) log.cc:844] Serialized 26 byte log entry
0116 02:59:18.753973 (+345935us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:20.014423 (+1260450us) log.cc:844] Serialized 14 byte log entry
0116 02:59:20.014705 (+ 282us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:20.014784 (+ 79us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:21.147921 (+1133137us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:21.527214 (+379293us) raft_consensus.cc:1680] finished
0116 02:59:21.603610 (+ 76396us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:21.636687 (+ 33077us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":201774,"prepare.queue_time_us":1197074,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":71,"raft.queue_time_us":176319,"raft.run_cpu_time_us":413,"raft.run_wall_time_us":49425,"spinlock_wait_cycles":455970048,"thread_start_us":1207341,"threads_started":2,"wal-append.queue_time_us":839971}
W20250116 02:59:22.220952 20748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1461) took 2275 ms (client timeout 2178 ms). Trace:
I20250116 02:59:22.230962 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.216244 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.241547 20756 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.242499 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.210268 22601 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:22.196707 20503 rpcz_store.cc:276] 0116 02:59:11.758262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:11.758765 (+ 503us) service_pool.cc:225] Handling call
0116 02:59:22.144340 (+10385575us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:19.606751 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:19.968111 (+361360us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:19.969518 (+ 1407us) write_op.cc:435] Acquired schema lock
0116 02:59:19.975067 (+ 5549us) tablet.cc:659] Decoding operations
0116 02:59:20.106222 (+131155us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:20.106298 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:20.106328 (+ 30us) tablet.cc:682] Acquiring locks for 3 operations
0116 02:59:20.106744 (+ 416us) tablet.cc:698] Row locks acquired
0116 02:59:20.106768 (+ 24us) write_op.cc:260] PREPARE: finished
0116 02:59:20.106960 (+ 192us) write_op.cc:270] Start()
0116 02:59:20.107095 (+ 135us) write_op.cc:276] Timestamp: P: 1736996360106901 usec, L: 0
0116 02:59:20.107114 (+ 19us) op_driver.cc:348] REPLICATION: starting
0116 02:59:20.107913 (+ 799us) log.cc:844] Serialized 1311 byte log entry
0116 02:59:21.121169 (+1013256us) op_driver.cc:464] REPLICATION: finished
0116 02:59:21.653394 (+532225us) write_op.cc:301] APPLY: starting
0116 02:59:21.659837 (+ 6443us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:21.670934 (+ 11097us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:21.731491 (+ 60557us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:21.790740 (+ 59249us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:21.822646 (+ 31906us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0116 02:59:21.822720 (+ 74us) write_op.cc:312] APPLY: finished
0116 02:59:21.823219 (+ 499us) log.cc:844] Serialized 35 byte log entry
0116 02:59:21.824862 (+ 1643us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:21.836351 (+ 11489us) write_op.cc:454] Released schema lock
0116 02:59:21.836433 (+ 82us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:21.837114 (+ 681us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:21.837156 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2291007,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":482608,"apply.run_cpu_time_us":4870,"apply.run_wall_time_us":199077,"num_ops":3,"prepare.queue_time_us":1233505,"prepare.run_cpu_time_us":16377,"prepare.run_wall_time_us":791345,"raft.queue_time_us":246554,"raft.run_cpu_time_us":16106,"raft.run_wall_time_us":369739,"replication_time_us":1013780,"spinlock_wait_cycles":1946368,"thread_start_us":1286678,"threads_started":4,"wal-append.queue_time_us":352579}]]}
I20250116 02:59:22.207782 22753 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.210001 22612 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:22.243366 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:22.205346 22764 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.247285 22715 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.234719 22600 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:22.227561 22749 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:22.250800 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 360) took 3583 ms (client timeout 4199 ms). Trace:
W20250116 02:59:22.299573 20752 rpcz_store.cc:269] 0116 02:59:18.667355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.199133 (+3531778us) service_pool.cc:225] Handling call
0116 02:59:22.250710 (+ 51577us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.252562 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.252919 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:22.253604 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1923) took 2056 ms. Trace:
I20250116 02:59:22.254317 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:22.143481 20504 catalog_manager.cc:5526] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 2 to 3, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 3 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:22.259243 20607 rpcz_store.cc:276] 0116 02:59:21.135834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.141342 (+ 5508us) service_pool.cc:225] Handling call
0116 02:59:22.200636 (+1059294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:21.915349 22824 consensus_queue.cc:237] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:22.264014 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 255276us
I20250116 02:59:22.262821 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 364) took 1968 ms. Trace:
I20250116 02:59:22.262650 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:22.263464 21335 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:22.263075 20757 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
W20250116 02:59:22.289589 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 65972.8us
I20250116 02:59:22.291234 21336 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:22.219180 21331 rpcz_store.cc:276] 0116 02:59:18.676886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.679993 (+ 3107us) service_pool.cc:225] Handling call
0116 02:59:18.680403 (+ 410us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:21.518735 (+2838332us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:21.766502 (+247767us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b5400592254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:21.766816 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5248872704}
W20250116 02:59:22.265244 22839 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.542s user 0.003s sys 0.011s
W20250116 02:59:22.234184 21330 rpcz_store.cc:269] 0116 02:59:18.875011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.171279 (+3296268us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:22.171423 (+ 144us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:22.166929 22801 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.229110 22690 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.301499 20605 rpcz_store.cc:276] 0116 02:59:20.197381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.226680 (+2029299us) service_pool.cc:225] Handling call
0116 02:59:22.251817 (+ 25137us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.252218 (+ 401us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.252249 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.252307 (+ 58us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:22.252335 (+ 28us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.252392 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.253517 (+ 1125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.302425 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 368) took 1352 ms. Trace:
I20250116 02:59:22.321100 20751 rpcz_store.cc:276] 0116 02:59:20.950138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.238159 (+1288021us) service_pool.cc:225] Handling call
0116 02:59:22.302359 (+ 64200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.303984 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 325) took 2055 ms. Trace:
I20250116 02:59:22.261569 20602 rpcz_store.cc:276] 0116 02:59:20.387219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.273744 (+886525us) service_pool.cc:225] Handling call
0116 02:59:22.224664 (+950920us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.306416 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:22.318228 20682 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:22.307379 20753 rpcz_store.cc:276] 0116 02:59:20.293955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.201217 (+1907262us) service_pool.cc:225] Handling call
0116 02:59:22.262741 (+ 61524us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.219815 22825 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:22.320902 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.308624 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:22.308800 22631 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:22.311111 21336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 476) took 3010 ms (client timeout 3056 ms). Trace:
W20250116 02:59:22.331336 21336 rpcz_store.cc:269] 0116 02:59:19.300468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.193601 (+2893133us) service_pool.cc:225] Handling call
0116 02:59:22.311039 (+117438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.300730 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:22.314710 20750 raft_consensus.cc:2388] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:22.315920 22753 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:22.263566 22796 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:22.296247 22600 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:22.322144 20603 rpcz_store.cc:276] 0116 02:59:20.241874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.229738 (+1987864us) service_pool.cc:225] Handling call
0116 02:59:22.262371 (+ 32633us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.297193 (+ 34822us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.297222 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.297271 (+ 49us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:22.297300 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.297361 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.297481 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.322898 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1862) took 3718 ms. Trace:
I20250116 02:59:22.344255 22631 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:22.344744 22796 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:22.289546 20748 rpcz_store.cc:269] 0116 02:59:19.943961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.206879 (+2262918us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:22.217019 (+ 10140us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:22.308209 21334 raft_consensus.cc:1235] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.323556 21333 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:22.329723 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.332737 20680 raft_consensus.cc:1235] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:22.333077 22821 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:22.317425 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:22.311856 22782 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:22.340353 20607 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.341590 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.343845 20749 rpcz_store.cc:276] 0116 02:59:18.597916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.304772 (+3706856us) service_pool.cc:225] Handling call
0116 02:59:22.315241 (+ 10469us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.316234 (+ 993us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.316263 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.316358 (+ 95us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:22.316400 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.316458 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.316570 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.316519 22633 consensus_queue.cc:1046] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20250116 02:59:22.304500 22790 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:22.351613 20606 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.034s sys 0.078s
I20250116 02:59:22.337293 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.357847 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:22.346300 22600 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:22.347087 20647 connection.cc:232] RPC call timeout handler was delayed by -2.1824628609999763s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2122s for a response
I20250116 02:59:22.363200 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:22.366317 20606 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.035s sys 0.078s
I20250116 02:59:22.352447 21329 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:22.366708 22633 consensus_queue.cc:1225] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer 57fcfcd5ab01438c9a4baf8a61871aa6 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250116 02:59:22.354176 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.362354 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1924) took 2163 ms. Trace:
I20250116 02:59:22.378705 20605 raft_consensus.cc:2405] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:22.364449 22665 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:22.391754 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 68878.4us
I20250116 02:59:22.387266 20753 raft_consensus.cc:1235] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.392501 20646 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:22.397560 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:22.371745 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.380077 22782 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.386888 20504 catalog_manager.cc:5526] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 2 to 3, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 3 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:22.385107 21328 raft_consensus.cc:2388] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:22.402820 20602 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:22.403529 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 327) took 2105 ms. Trace:
I20250116 02:59:22.381127 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:22.376495 22355 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:22.361836 22821 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.396073 21336 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 5. Ops: []
I20250116 02:59:22.384747 22801 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:22.402040 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:22.383759 22755 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:22.404534 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:22.404522 21335 raft_consensus.cc:1235] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.403810 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.408154 20682 raft_consensus.cc:3049] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:22.406471 20571 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:22.411192 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 707) took 3702 ms. Trace:
I20250116 02:59:22.418581 20682 raft_consensus.cc:738] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:22.420796 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 471) took 3652 ms. Trace:
I20250116 02:59:22.421231 21335 rpcz_store.cc:276] 0116 02:59:18.761915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.370010 (+3608095us) service_pool.cc:225] Handling call
0116 02:59:22.397133 (+ 27123us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.413783 (+ 16650us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.413972 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.411427 21300 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:22.405253 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1865) took 3799 ms. Trace:
I20250116 02:59:22.423053 20753 rpcz_store.cc:276] 0116 02:59:18.611885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.332558 (+3720673us) service_pool.cc:225] Handling call
0116 02:59:22.367833 (+ 35275us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.398352 (+ 30519us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.398553 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.393407 22601 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:22.393175 22612 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:22.419462 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 464) took 3702 ms. Trace:
I20250116 02:59:22.417253 20749 raft_consensus.cc:1235] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:22.424566 22631 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:22.424769 20682 consensus_queue.cc:260] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.401553 22764 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:22.429180 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.428493 21336 rpcz_store.cc:276] 0116 02:59:18.717694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.336913 (+3619219us) service_pool.cc:225] Handling call
0116 02:59:22.366112 (+ 29199us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.419215 (+ 53103us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.419405 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.429464 20676 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.430637 20752 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.428519 20607 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:22.403873 22782 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:22.404043 22747 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:22.435742 22753 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:22.442888 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.437167 22749 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:22.436297 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.425277 22790 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:22.436481 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 46873.5us
I20250116 02:59:22.434767 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1867) took 3822 ms. Trace:
I20250116 02:59:22.420217 21327 rpcz_store.cc:276] 0116 02:59:18.709156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.336794 (+3627638us) service_pool.cc:225] Handling call
0116 02:59:22.377833 (+ 41039us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.392979 (+ 15146us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.395768 (+ 2789us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.401205 (+ 5437us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:22.410857 (+ 9652us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.410955 (+ 98us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.411106 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.451175 20749 rpcz_store.cc:276] 0116 02:59:18.612538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.365892 (+3753354us) service_pool.cc:225] Handling call
0116 02:59:22.385810 (+ 19918us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.434473 (+ 48663us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.434709 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.379097 20599 rpcz_store.cc:276] 0116 02:59:20.197705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.221019 (+2023314us) service_pool.cc:225] Handling call
0116 02:59:22.299255 (+ 78236us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.338027 (+ 38772us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.339220 (+ 1193us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.341489 (+ 2269us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:22.350818 (+ 9329us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.355083 (+ 4265us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.361314 (+ 6231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.468530 20605 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:22.469465 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.425639 22715 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:22.460280 20603 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:22.436475 22519 raft_consensus.cc:3049] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:22.473039 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.433207 22665 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:22.403861 20602 rpcz_store.cc:276] 0116 02:59:20.297870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.337499 (+2039629us) service_pool.cc:225] Handling call
0116 02:59:22.370711 (+ 33212us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.403256 (+ 32545us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.403468 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.435273 22766 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:22.447132 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.487809 21331 raft_consensus.cc:1235] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.471719 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:22.494562 20753 raft_consensus.cc:1235] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:22.497365 20675 raft_consensus.cc:3049] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:22.497804 20675 raft_consensus.cc:738] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:22.498793 20675 consensus_queue.cc:260] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.472544 22690 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:22.477387 22519 raft_consensus.cc:738] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:22.480990 22796 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:22.503806 21327 raft_consensus.cc:1235] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.509065 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.510529 20680 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.501231 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1870) took 3887 ms. Trace:
I20250116 02:59:22.511214 20753 rpcz_store.cc:276] 0116 02:59:18.613311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.427265 (+3813954us) service_pool.cc:225] Handling call
0116 02:59:22.483950 (+ 56685us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.501016 (+ 17066us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.501183 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.508026 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1928) took 1366 ms. Trace:
I20250116 02:59:22.480983 22665 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:22.512039 20607 rpcz_store.cc:276] 0116 02:59:21.134970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.484798 (+1349828us) service_pool.cc:225] Handling call
0116 02:59:22.492935 (+ 8137us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.501167 (+ 8232us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.501202 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.501254 (+ 52us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:22.501288 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.501354 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.501466 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.479408 22825 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:22.517268 20649 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:22.525322 20604 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:22.500149 20675 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:22.518458 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.526947 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1871) took 3913 ms. Trace:
I20250116 02:59:22.527531 20674 raft_consensus.cc:2388] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:22.527366 20749 rpcz_store.cc:276] 0116 02:59:18.613551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.469062 (+3855511us) service_pool.cc:225] Handling call
0116 02:59:22.508419 (+ 39357us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.526588 (+ 18169us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.526638 (+ 50us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.526681 (+ 43us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:22.526711 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.526767 (+ 56us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.526891 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.533169 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.534109 20607 raft_consensus.cc:2388] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:22.537165 20649 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:22.510059 22821 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:22.493148 22631 consensus_queue.cc:1046] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20250116 02:59:22.540685 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 714) took 1006 ms. Trace:
I20250116 02:59:22.523124 22665 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:22.543262 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 181004us
I20250116 02:59:22.542986 21336 rpcz_store.cc:276] 0116 02:59:21.534576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.480456 (+945880us) service_pool.cc:225] Handling call
0116 02:59:22.493035 (+ 12579us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.536964 (+ 43929us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.536997 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.540419 (+ 3422us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:22.540460 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.540523 (+ 63us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.540631 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:22.544216 20757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1860) took 3946 ms (client timeout 2398 ms). Trace:
W20250116 02:59:22.545295 20757 rpcz_store.cc:269] 0116 02:59:18.597397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.527511 (+1930114us) service_pool.cc:225] Handling call
0116 02:59:22.540110 (+2012599us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b5400396554. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:22.544141 (+ 4031us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1657566592}
I20250116 02:59:22.532856 20599 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.528401 22843 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:22.550448 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.537727 20748 raft_consensus.cc:1235] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:22.550410 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:22.579316 21332 raft_consensus.cc:2388] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:22.539002 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.539252 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 717) took 1003 ms. Trace:
I20250116 02:59:22.554184 20749 raft_consensus.cc:1235] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:22.555593 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:22.549968 22850 log.cc:926] Time spent T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.078s user 0.000s sys 0.002s
W20250116 02:59:22.565081 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 84377.3us
I20250116 02:59:22.566500 20753 raft_consensus.cc:1235] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.545735 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1927) took 1416 ms. Trace:
I20250116 02:59:22.547533 22791 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:22.577203 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1929) took 1441 ms. Trace:
I20250116 02:59:22.586171 20607 rpcz_store.cc:276] 0116 02:59:21.135407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.539160 (+1403753us) service_pool.cc:225] Handling call
0116 02:59:22.546572 (+ 7412us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.576837 (+ 30265us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.576868 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.576918 (+ 50us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:22.576954 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.577018 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.577153 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.583341 20649 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:22.551569 22755 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:22.580403 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.575103 22631 consensus_queue.cc:1225] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Peer 57fcfcd5ab01438c9a4baf8a61871aa6 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250116 02:59:22.580387 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1598) took 3923 ms. Trace:
I20250116 02:59:22.592665 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.581167 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.585561 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1444) took 3746 ms. Trace:
I20250116 02:59:22.595211 20646 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:22.596202 20753 rpcz_store.cc:276] 0116 02:59:18.839279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.524775 (+3685496us) service_pool.cc:225] Handling call
0116 02:59:22.566282 (+ 41507us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.585269 (+ 18987us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.585498 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.597929 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1451) took 3468 ms. Trace:
I20250116 02:59:22.598249 20749 rpcz_store.cc:276] 0116 02:59:19.125156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.538890 (+3413734us) service_pool.cc:225] Handling call
0116 02:59:22.553983 (+ 15093us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.586343 (+ 32360us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.593059 (+ 6716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.574745 22825 consensus_queue.cc:1046] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250116 02:59:22.601419 22825 consensus_queue.cc:1225] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer 57fcfcd5ab01438c9a4baf8a61871aa6 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250116 02:59:22.567838 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 711) took 2554 ms. Trace:
I20250116 02:59:22.592072 20600 rpcz_store.cc:276] 0116 02:59:21.134182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.401575 (+1267393us) service_pool.cc:225] Handling call
0116 02:59:22.469623 (+ 68048us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.509686 (+ 40063us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.511036 (+ 1350us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.524795 (+ 13759us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:22.532251 (+ 7456us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.537442 (+ 5191us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.543012 (+ 5570us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.602975 20600 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43854
I20250116 02:59:22.592511 22755 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:22.603684 20750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1442) took 3765 ms (client timeout 1770 ms). Trace:
I20250116 02:59:22.604022 22755 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:22.604730 21331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 473) took 3687 ms. Trace:
I20250116 02:59:22.605108 21331 rpcz_store.cc:276] 0116 02:59:18.917645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.444050 (+3526405us) service_pool.cc:225] Handling call
0116 02:59:22.444429 (+ 379us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.488333 (+ 43904us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.604462 (+116129us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b54004df3d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:22.604650 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99131648}
I20250116 02:59:22.606870 20755 raft_consensus.cc:1235] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:22.607522 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 712) took 2517 ms. Trace:
I20250116 02:59:22.607916 21335 rpcz_store.cc:276] 0116 02:59:20.089867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.570321 (+2480454us) service_pool.cc:225] Handling call
0116 02:59:22.606910 (+ 36589us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.607183 (+ 273us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.607212 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.607257 (+ 45us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:22.607315 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.607365 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.607458 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.609071 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 698) took 3932 ms. Trace:
I20250116 02:59:22.609447 21335 raft_consensus.cc:1235] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.610834 22796 consensus_queue.cc:1046] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:59:22.585368 21327 rpcz_store.cc:276] 0116 02:59:21.535626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.502976 (+967350us) service_pool.cc:225] Handling call
0116 02:59:22.503520 (+ 544us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.518891 (+ 15371us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.537458 (+ 18567us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.611853 22796 raft_consensus.cc:3049] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:22.612296 21327 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.610109 21328 rpcz_store.cc:276] 0116 02:59:20.025774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.482766 (+2456992us) service_pool.cc:225] Handling call
0116 02:59:22.544033 (+ 61267us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.555275 (+ 11242us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.557031 (+ 1756us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.567463 (+ 10432us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:22.567540 (+ 77us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.567617 (+ 77us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.567771 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.601998 22825 raft_consensus.cc:3049] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:22.613787 22825 raft_consensus.cc:738] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:22.609473 20753 raft_consensus.cc:1235] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.609477 21334 rpcz_store.cc:276] 0116 02:59:18.676572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:18.677669 (+ 1097us) service_pool.cc:225] Handling call
0116 02:59:18.677989 (+ 320us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.353855 (+3675866us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.608791 (+254936us) spinlock_profiling.cc:243] Waited 3.61 s on lock 0x7b54006c5a54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:22.609010 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8303853824}
I20250116 02:59:22.612228 22796 raft_consensus.cc:738] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:22.616381 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.617219 20604 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:22.618136 20682 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.618856 22755 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:22.621410 22755 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:22.608481 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1861) took 4010 ms. Trace:
I20250116 02:59:22.595741 20608 rpcz_store.cc:276] 0116 02:59:18.656697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.603358 (+2946661us) service_pool.cc:225] Handling call
0116 02:59:22.145918 (+542560us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.146323 (+ 405us) raft_consensus.cc:1529] Early marking committed up to index 3
0116 02:59:22.472478 (+326155us) log.cc:844] Serialized 14 byte log entry
0116 02:59:22.579366 (+106888us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.579428 (+ 62us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:22.579484 (+ 56us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.580180 (+ 696us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.580318 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":329700,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":520,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":154,"thread_start_us":407870,"threads_started":2,"wal-append.queue_time_us":102305}
I20250116 02:59:22.615409 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1454) took 3245 ms. Trace:
I20250116 02:59:22.599038 20649 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:22.610040 20571 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:22.600451 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 363) took 3299 ms. Trace:
I20250116 02:59:22.649695 21332 raft_consensus.cc:2388] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
W20250116 02:59:22.623035 20750 rpcz_store.cc:269] 0116 02:59:18.838712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:20.099884 (+1261172us) service_pool.cc:225] Handling call
0116 02:59:22.603417 (+2503533us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b5400043b54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:22.603592 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4855291776}
I20250116 02:59:22.688974 20603 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:22.689785 20607 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:22.690164 20750 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:22.602917 21330 raft_consensus.cc:1235] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.690897 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 365) took 2396 ms. Trace:
I20250116 02:59:22.691151 21330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 708) took 3981 ms. Trace:
I20250116 02:59:22.691253 20750 rpcz_store.cc:276] 0116 02:59:20.294381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.689114 (+2394733us) service_pool.cc:225] Handling call
0116 02:59:22.689607 (+ 493us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.690588 (+ 981us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.690810 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.645115 22519 consensus_queue.cc:260] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:22.667213 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 72118.8us
I20250116 02:59:22.682196 21302 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:22.693373 21302 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:22.694357 22748 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:22.694885 22748 raft_consensus.cc:695] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:22.661804 22825 consensus_queue.cc:260] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.696377 22825 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.697014 22749 raft_consensus.cc:2743] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:22.667390 20677 raft_consensus.cc:3049] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:22.697530 20677 raft_consensus.cc:738] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:22.697310 22600 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.692260 21330 rpcz_store.cc:276] 0116 02:59:18.709409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.342267 (+3632858us) service_pool.cc:225] Handling call
0116 02:59:22.602428 (+260161us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:22.602702 (+ 274us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.690850 (+ 88148us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.691061 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5958016}
I20250116 02:59:22.644976 22791 raft_consensus.cc:695] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:22.663709 22530 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:22.699110 22530 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:22.699503 22530 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:22.667081 22633 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:22.700088 22633 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:22.700502 22633 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:22.603725 20600 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:59:22.691663 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1932) took 1020 ms. Trace:
I20250116 02:59:22.701987 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:22.702299 20607 rpcz_store.cc:276] 0116 02:59:21.671435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.596004 (+924569us) service_pool.cc:225] Handling call
0116 02:59:22.689564 (+ 93560us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.690169 (+ 605us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.691478 (+ 1309us) spinlock_profiling.cc:243] Waited 82.6 ms on lock 0x7b5400290ed0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:22.691605 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":189953792}
I20250116 02:59:22.704126 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:22.705349 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.705976 20607 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:22.695978 22748 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.624189 20755 rpcz_store.cc:276] 0116 02:59:18.597655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.301398 (+3703743us) service_pool.cc:225] Handling call
0116 02:59:22.606687 (+305289us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.607302 (+ 615us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.608281 (+ 979us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b5400043b50. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:22.608398 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":671965568}
I20250116 02:59:22.717139 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 366) took 2418 ms. Trace:
I20250116 02:59:22.595395 20602 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:22.664395 21336 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.706151 20680 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:22.702536 20749 raft_consensus.cc:1270] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:59:22.719779 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1463) took 2621 ms. Trace:
I20250116 02:59:22.720203 20749 rpcz_store.cc:276] 0116 02:59:20.098302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.598817 (+2500515us) service_pool.cc:225] Handling call
0116 02:59:22.702125 (+103308us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.719538 (+ 17413us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.719726 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.721168 20676 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:22.687466 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.662241 20753 rpcz_store.cc:276] 0116 02:59:19.370372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.608733 (+3238361us) service_pool.cc:225] Handling call
0116 02:59:22.609189 (+ 456us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.615148 (+ 5959us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.615348 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.698485 20677 consensus_queue.cc:260] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.717509 20755 rpcz_store.cc:276] 0116 02:59:20.298457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.716090 (+2417633us) service_pool.cc:225] Handling call
0116 02:59:22.716569 (+ 479us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.716796 (+ 227us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.716823 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.716862 (+ 39us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:22.716898 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.716947 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.717052 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:22.707532 22823 consensus_meta.cc:287] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.509s user 0.000s sys 0.010s
I20250116 02:59:22.699656 22791 consensus_queue.cc:237] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.709560 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.710737 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.616380 22796 consensus_queue.cc:260] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:22.711495 22519 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:22.711841 22821 consensus_queue.cc:1046] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:59:22.712059 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:22.701938 20600 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:22.714289 22755 consensus_queue.cc:1046] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 02:59:22.706761 20603 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.618304 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:22.777766 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 397306us
I20250116 02:59:22.627540 21335 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:22.651032 20605 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:22.649500 20757 rpcz_store.cc:276] 0116 02:59:19.301210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.550095 (+3248885us) service_pool.cc:225] Handling call
0116 02:59:22.561710 (+ 11615us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.600191 (+ 38481us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.600220 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.600260 (+ 40us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:22.600285 (+ 25us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.600328 (+ 43us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.600406 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.739663 20756 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:22.739605 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 719) took 1203 ms. Trace:
I20250116 02:59:22.779148 21336 rpcz_store.cc:276] 0116 02:59:21.536353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.623079 (+1086726us) service_pool.cc:225] Handling call
0116 02:59:22.664179 (+ 41100us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.739245 (+ 75066us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.739527 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:22.744194 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 145199us
I20250116 02:59:22.749145 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.750294 20752 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:22.753042 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:22.754664 20749 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:59:22.756376 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.754637 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:22.782974 20607 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:22.759235 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 149288us
I20250116 02:59:22.759848 20677 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.784191 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 376) took 1197 ms. Trace:
I20250116 02:59:22.782194 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 369) took 1831 ms. Trace:
I20250116 02:59:22.784792 20749 rpcz_store.cc:276] 0116 02:59:20.950949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.720955 (+1770006us) service_pool.cc:225] Handling call
0116 02:59:22.747339 (+ 26384us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.781879 (+ 34540us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.782128 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.760879 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.765450 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 371) took 1806 ms. Trace:
W20250116 02:59:22.769860 22855 log.cc:926] Time spent T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.068s user 0.000s sys 0.003s
I20250116 02:59:22.770730 22782 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250116 02:59:22.770989 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:22.794850 21300 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:22.795324 20572 leader_election.cc:336] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.098s (SENT)
W20250116 02:59:22.796025 20572 leader_election.cc:336] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.178s (SENT)
I20250116 02:59:22.721886 22815 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:22.796908 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 370) took 1845 ms. Trace:
I20250116 02:59:22.797756 21302 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:22.771519 22631 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:22.786049 20755 rpcz_store.cc:276] 0116 02:59:20.958806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.727661 (+1768855us) service_pool.cc:225] Handling call
0116 02:59:22.757959 (+ 30298us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.762213 (+ 4254us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.762248 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.762298 (+ 50us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:22.765219 (+ 2921us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.765287 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.765395 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58,"raft.queue_time_us":7616,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111}
I20250116 02:59:22.728715 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1441) took 3819 ms. Trace:
I20250116 02:59:22.758412 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1601) took 4078 ms. Trace:
I20250116 02:59:22.771539 22600 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:22.787170 20678 raft_consensus.cc:2405] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:22.789136 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 367) took 2387 ms. Trace:
I20250116 02:59:22.809926 20750 rpcz_store.cc:276] 0116 02:59:20.401852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.719483 (+2317631us) service_pool.cc:225] Handling call
0116 02:59:22.776652 (+ 57169us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.788789 (+ 12137us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.788819 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.788863 (+ 44us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:22.788895 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.788958 (+ 63us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.789065 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.792297 20757 raft_consensus.cc:1235] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
W20250116 02:59:22.803244 20647 leader_election.cc:336] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.576s (SENT)
I20250116 02:59:22.811312 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 372) took 1765 ms. Trace:
I20250116 02:59:22.784570 20756 rpcz_store.cc:276] 0116 02:59:21.581924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.204738 (+622814us) service_pool.cc:225] Handling call
0116 02:59:22.779104 (+574366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.811662 20757 rpcz_store.cc:276] 0116 02:59:21.046421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.779190 (+1732769us) service_pool.cc:225] Handling call
0116 02:59:22.792068 (+ 12878us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.811044 (+ 18976us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.811257 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.813107 20756 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:22.811406 20647 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:22.813107 20757 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:22.747902 22823 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:22.810719 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1934) took 1116 ms. Trace:
I20250116 02:59:22.827209 20605 rpcz_store.cc:276] 0116 02:59:21.694861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.785998 (+1091137us) service_pool.cc:225] Handling call
0116 02:59:22.788200 (+ 2202us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.810303 (+ 22103us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.810334 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.810389 (+ 55us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:22.810422 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.810490 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.810648 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.821424 20571 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:22.781946 22846 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:22.804234 20601 rpcz_store.cc:276] 0116 02:59:18.676242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.591366 (+2915124us) service_pool.cc:225] Handling call
0116 02:59:21.679474 (+ 88108us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:22.518571 (+839097us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b18003d8220. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc32699a08c 00007fc323db8bfc 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd3305 00007fc326d073d0 00007fc326d07f6b 00007fc326d1278b
0116 02:59:22.524869 (+ 6298us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.524898 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:22.544839 (+ 19941us) log.cc:844] Serialized 26 byte log entry
0116 02:59:22.751921 (+207082us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:22.751961 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.752034 (+ 73us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:22.754062 (+ 2028us) raft_consensus.cc:1680] finished
0116 02:59:22.754089 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.754343 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":442152,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":7788,"spinlock_wait_cycles":5112192,"thread_start_us":604374,"threads_started":2,"wal-append.queue_time_us":196557}
I20250116 02:59:22.840346 20604 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:22.844996 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1935) took 1115 ms. Trace:
I20250116 02:59:22.845501 20608 rpcz_store.cc:276] 0116 02:59:21.732048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.796562 (+1064514us) service_pool.cc:225] Handling call
0116 02:59:22.831747 (+ 35185us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.844597 (+ 12850us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.844631 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.844687 (+ 56us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:22.844719 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.844788 (+ 69us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.844926 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.848598 20607 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:22.856453 22796 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.808820 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.875088 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 373) took 1809 ms. Trace:
I20250116 02:59:22.820827 22660 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:22.883986 20752 rpcz_store.cc:276] 0116 02:59:21.065798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.780794 (+1714996us) service_pool.cc:225] Handling call
0116 02:59:22.781137 (+ 343us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.855792 (+ 74655us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.855823 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.874749 (+ 18926us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:22.874800 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.874877 (+ 77us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.875018 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:22.831013 22690 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:22.885710 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 48877.6us
I20250116 02:59:22.886991 20755 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:22.839073 22766 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:22.830631 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:22.892108 20571 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:22.884193 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 88300.1us
I20250116 02:59:22.803963 22665 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.809201 22518 consensus_queue.cc:1046] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20250116 02:59:22.894501 20608 raft_consensus.cc:1270] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:22.874287 20754 raft_consensus.cc:1235] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:22.803961 20748 rpcz_store.cc:276] 0116 02:59:18.838360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.477352 (+3638992us) service_pool.cc:225] Handling call
0116 02:59:22.528936 (+ 51584us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.591629 (+ 62693us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.657435 (+ 65806us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:22.906055 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 79982.6us
I20250116 02:59:22.892031 22766 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:22.895208 22660 raft_consensus.cc:695] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:22.963375 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 47444.2us
I20250116 02:59:22.983676 22863 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:22.984596 22863 raft_consensus.cc:3049] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:22.984942 22863 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:22.909010 22782 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:22.910197 22800 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250116 02:59:22.985915 22863 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:22.988580 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:22.989934 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:22.991351 22600 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:22.992168 22600 consensus_queue.cc:1035] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:59:22.994143 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:22.994423 21300 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:22.995296 22782 raft_consensus.cc:2798] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
W20250116 02:59:22.967674 20720 connection.cc:232] RPC call timeout handler was delayed by -5.6710337250000293s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
I20250116 02:59:22.971781 22823 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:22.972328 22749 consensus_queue.cc:1046] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250116 02:59:22.910115 22690 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:22.982303 21329 consensus_meta.cc:287] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.611s user 0.005s sys 0.005s
I20250116 02:59:23.042203 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:22.974915 22857 log.cc:926] Time spent T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20250116 02:59:22.873260 22748 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20250116 02:59:22.992380 22518 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:22.917949 22766 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:22.971043 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.970278 22815 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:22.917018 20603 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:23.005045 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:23.064205 20604 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
W20250116 02:59:23.005673 20720 connection.cc:232] RPC call timeout handler was delayed by -4.8847269139998843s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
I20250116 02:59:22.973225 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.976181 20599 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:23.040959 22868 consensus_queue.cc:1046] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:22.988818 22863 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:23.042449 22665 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:23.042677 22792 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:22.973436 22860 log.cc:926] Time spent T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250116 02:59:23.048534 20680 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:22.993290 22660 consensus_queue.cc:237] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.049685 20607 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:23.042438 22690 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:23.054286 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 196378us
I20250116 02:59:23.080027 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:22.908351 20602 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 1.
I20250116 02:59:23.052620 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.064419 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:23.064502 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 93586.5us
W20250116 02:59:23.066406 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 268737us
I20250116 02:59:23.067629 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.069811 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.070463 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.066516 22843 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:23.077976 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 86676us
I20250116 02:59:23.091782 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:23.080724 20646 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.083287 21302 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.044860 21329 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:23.097004 20605 raft_consensus.cc:2388] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
W20250116 02:59:23.053697 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 47047.3us
I20250116 02:59:23.091364 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:23.076594 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 273366us
W20250116 02:59:23.084901 20723 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.918s (SENT) [suppressed 50 similar messages]
I20250116 02:59:23.101078 20649 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:23.102382 20601 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 3
I20250116 02:59:23.102388 20571 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:23.103403 20720 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:23.109755 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:23.113175 20647 leader_election.cc:336] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.945s (SENT)
I20250116 02:59:23.113765 20647 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:23.115895 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.114432 20604 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20250116 02:59:23.114987 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.088564 22782 raft_consensus.cc:695] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:23.092219 22631 raft_consensus.cc:695] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:23.100003 22665 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:23.094033 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.116851 20604 pending_rounds.cc:77] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f: Aborting all ops after (but not including) 5
I20250116 02:59:23.104530 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.125315 20649 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:23.077343 22518 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:23.104135 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 721) took 1470 ms. Trace:
I20250116 02:59:23.121156 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.122431 20681 raft_consensus.cc:2388] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:23.081424 22792 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:23.126360 20604 pending_rounds.cc:99] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250116 02:59:23.113404 22867 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:23.131498 20604 raft_consensus.cc:2883] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:23.134447 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:23.136409 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 51181.9us
I20250116 02:59:23.123406 22690 raft_consensus.cc:2743] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:23.123154 22815 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:23.139564 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.167697 20683 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
W20250116 02:59:23.170554 20572 leader_election.cc:336] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.513s (SENT)
I20250116 02:59:23.171479 20572 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:23.139256 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 374) took 2030 ms. Trace:
I20250116 02:59:23.123389 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:23.174998 21330 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:23.175607 20748 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2"
dest_uuid: "17446c8dacf04461acb843547665250f"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:42186
I20250116 02:59:23.176651 20748 raft_consensus.cc:604] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.181399 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1868) took 4568 ms. Trace:
W20250116 02:59:23.181916 22877 log.cc:926] Time spent T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250116 02:59:23.153509 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.183066 20681 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:23.157519 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.158346 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.158993 22631 consensus_queue.cc:237] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.163071 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1600) took 4487 ms. Trace:
I20250116 02:59:23.164078 20571 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:23.130097 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.131587 21329 rpcz_store.cc:276] 0116 02:59:21.633617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.300331 (+666714us) service_pool.cc:225] Handling call
0116 02:59:23.099437 (+799106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:23.166918 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 53819.6us
I20250116 02:59:23.173439 20749 rpcz_store.cc:276] 0116 02:59:21.109456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.793784 (+1684328us) service_pool.cc:225] Handling call
0116 02:59:22.807253 (+ 13469us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.835967 (+ 28714us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:23.074117 (+238150us) log.cc:844] Serialized 14 byte log entry
0116 02:59:23.138940 (+ 64823us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:23.138991 (+ 51us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:23.139025 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:23.139095 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:23.139208 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13798,"prepare.queue_time_us":288520,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":52990,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":35665,"spinlock_wait_cycles":106368,"thread_start_us":219031,"threads_started":2}
I20250116 02:59:23.158475 22748 raft_consensus.cc:3049] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:23.133450 22867 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:23.199435 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 203532us
I20250116 02:59:23.162233 22825 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20250116 02:59:23.131006 22864 raft_consensus.cc:2743] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:23.200670 20606 rpcz_store.cc:276] 0116 02:59:18.675689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:21.581795 (+2906106us) service_pool.cc:225] Handling call
0116 02:59:21.660846 (+ 79051us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:22.370069 (+709223us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.370107 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:22.387875 (+ 17768us) log.cc:844] Serialized 26 byte log entry
0116 02:59:22.725471 (+337596us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:22.725506 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.725572 (+ 66us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:23.162804 (+437232us) raft_consensus.cc:1680] finished
0116 02:59:23.162822 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:23.162988 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1662,"raft.queue_time_us":531427,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":12627,"spinlock_wait_cycles":1792,"thread_start_us":994560,"threads_started":2,"wal-append.queue_time_us":438675}
I20250116 02:59:23.208516 20681 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:23.208724 21330 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:23.209015 20674 raft_consensus.cc:2388] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:23.209389 22856 log.cc:926] Time spent T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.304s user 0.003s sys 0.000s
I20250116 02:59:23.209904 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:23.211210 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.213508 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.196027 22690 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:23.244342 20647 leader_election.cc:336] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.024s (SENT)
I20250116 02:59:23.128947 22791 consensus_queue.cc:1046] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20250116 02:59:23.215456 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:23.245955 20677 raft_consensus.cc:2388] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:23.213275 20646 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:23.172168 22665 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:23.250770 22665 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:23.251194 22665 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:23.145071 22749 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20250116 02:59:23.198223 22865 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:23.131163 22633 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:23.236524 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 72526.1us
I20250116 02:59:23.181918 20754 rpcz_store.cc:276] 0116 02:59:18.612796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.372241 (+3759445us) service_pool.cc:225] Handling call
0116 02:59:22.429117 (+ 56876us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.909086 (+479969us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:23.181020 (+271934us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b540033dbd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:23.181303 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":926906240}
I20250116 02:59:23.238374 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.215206 22748 raft_consensus.cc:738] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:23.228950 22530 consensus_meta.cc:287] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.529s user 0.008s sys 0.004s
I20250116 02:59:23.243644 20604 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:23.247749 20720 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:23.248260 22873 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.119s
I20250116 02:59:23.248703 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1863) took 4650 ms. Trace:
I20250116 02:59:23.249794 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.248286 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.253787 20647 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.228499 22867 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:23.236205 22519 consensus_queue.cc:1035] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250116 02:59:23.279013 22748 consensus_queue.cc:260] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.262964 20600 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:23.264140 22633 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:23.269104 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 185880us
I20250116 02:59:23.271085 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.276146 20599 pending_rounds.cc:77] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Aborting all ops after (but not including) 5
I20250116 02:59:23.277506 22766 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:23.262748 22878 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:23.299105 20608 raft_consensus.cc:1235] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
W20250116 02:59:23.286898 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 116401us
I20250116 02:59:23.268409 20751 rpcz_store.cc:276] 0116 02:59:18.598184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.337986 (+3739802us) service_pool.cc:225] Handling call
0116 02:59:22.357106 (+ 19120us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:22.400264 (+ 43158us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.400308 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:22.400916 (+ 608us) log.cc:844] Serialized 26 byte log entry
0116 02:59:22.611217 (+210301us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b18003c3fa0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32698be4b 00007fc326991913 00007fc326991cc1 00007fc326ce345f
0116 02:59:22.739973 (+128756us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:22.780332 (+ 40359us) log.cc:844] Serialized 14 byte log entry
0116 02:59:22.780559 (+ 227us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.780627 (+ 68us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:23.233768 (+453141us) raft_consensus.cc:1680] finished
0116 02:59:23.248411 (+ 14643us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:23.248635 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15904,"prepare.queue_time_us":41325,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":5859,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":28454,"spinlock_wait_cycles":13288064,"thread_start_us":127500,"threads_started":1,"wal-append.queue_time_us":484473}
I20250116 02:59:23.270700 22530 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:23.292786 21302 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:23.228403 22782 consensus_queue.cc:237] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.289568 20606 raft_consensus.cc:3049] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:23.304401 20606 raft_consensus.cc:738] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:23.295504 20683 raft_consensus.cc:2363] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:23.304668 20571 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:23.287912 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 74744.5us
I20250116 02:59:23.283004 20602 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:23.298377 20599 pending_rounds.cc:99] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250116 02:59:23.298183 20679 raft_consensus.cc:2388] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:23.306550 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 72470.6us
I20250116 02:59:23.306547 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:23.306003 20646 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:23.308851 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 61303.8us
I20250116 02:59:23.322993 21332 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:23.286893 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 102759us
I20250116 02:59:23.298627 22800 raft_consensus.cc:2798] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
W20250116 02:59:23.329391 22888 log.cc:926] Time spent T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20250116 02:59:23.335861 20601 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:23.300451 22715 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:23.340824 20682 raft_consensus.cc:2388] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:23.300737 22823 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:23.288177 22660 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
W20250116 02:59:23.347465 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 45940us
I20250116 02:59:23.341861 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:23.344583 20647 connection.cc:232] RPC call timeout handler was delayed by -1.9234639280000465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2398s for a response
I20250116 02:59:23.339371 21299 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:23.348717 20601 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 3
W20250116 02:59:23.348912 20647 connection.cc:232] RPC call timeout handler was delayed by -1.7747822320000068s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2888s for a response
W20250116 02:59:23.349804 20647 connection.cc:232] RPC call timeout handler was delayed by -1.7198628610000242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2636s for a response
W20250116 02:59:23.350129 20647 connection.cc:232] RPC call timeout handler was delayed by -1.1845723240001007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31820000000000004s for a response
I20250116 02:59:23.319954 22864 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:23.322853 20599 raft_consensus.cc:2883] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:23.331422 20600 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:23.319730 22878 raft_consensus.cc:695] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:23.357004 20604 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:23.357877 20606 consensus_queue.cc:260] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.365636 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:23.343927 22690 raft_consensus.cc:2743] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250116 02:59:23.371167 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 65238.4us
I20250116 02:59:23.345008 22792 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:23.376188 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 69364.2us
W20250116 02:59:23.376870 20572 leader_election.cc:336] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.670s (SENT)
I20250116 02:59:23.377247 20572 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:23.378024 20606 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:23.360284 22881 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:23.351068 22518 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:23.385397 20571 leader_election.cc:400] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4.
I20250116 02:59:23.385799 20571 leader_election.cc:403] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250116 02:59:23.388911 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.396273 20604 raft_consensus.cc:2388] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:23.420285 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 43981.6us
I20250116 02:59:23.368676 22800 raft_consensus.cc:695] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:23.421487 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 57095.4us
W20250116 02:59:23.424113 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 47274.7us
I20250116 02:59:23.425436 20683 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:23.426081 20683 raft_consensus.cc:604] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:23.387893 21299 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.426739 20683 raft_consensus.cc:612] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:23.380363 22600 raft_consensus.cc:2743] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:23.430661 20755 raft_consensus.cc:1235] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250116 02:59:23.432559 22886 log.cc:926] Time spent T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20250116 02:59:23.386195 22873 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:23.384981 22864 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:23.436070 22864 raft_consensus.cc:695] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:23.437018 22864 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.417232 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:43854 (request call id 1933) took 1689 ms. Trace:
I20250116 02:59:23.438930 20600 rpcz_store.cc:276] 0116 02:59:21.678843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.602790 (+923947us) service_pool.cc:225] Handling call
0116 02:59:23.367863 (+765073us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":394}
I20250116 02:59:23.440457 20600 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:23.434593 22800 consensus_queue.cc:237] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.422647 20601 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:23.427806 20572 leader_election.cc:336] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.700s (SENT)
I20250116 02:59:23.429133 22884 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20250116 02:59:23.430034 22766 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:23.452008 21332 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:23.466058 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 47003.5us
I20250116 02:59:23.424320 22825 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4.
I20250116 02:59:23.423184 22881 raft_consensus.cc:695] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:23.467392 20646 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:23.469291 20681 raft_consensus.cc:1235] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:23.474867 20601 raft_consensus.cc:1235] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:23.454222 22873 raft_consensus.cc:695] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:23.485070 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.497397 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.550354 20681 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:23.550340 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 257606us
W20250116 02:59:23.550673 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 59633.2us
I20250116 02:59:23.482951 20679 raft_consensus.cc:1235] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:23.479177 22874 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
W20250116 02:59:23.552035 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 61848us
I20250116 02:59:23.562551 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.563411 21302 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:23.568331 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.572260 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.576843 20720 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.584070 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:23.584288 22881 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.585505 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.585706 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.488814 22891 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20250116 02:59:23.586544 20601 raft_consensus.cc:2388] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:23.558967 21330 raft_consensus.cc:1235] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
W20250116 02:59:23.592733 20571 connection.cc:232] RPC call timeout handler was delayed by -1.6566578540000592s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
I20250116 02:59:23.551604 22530 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:23.490338 22867 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:23.597292 20571 connection.cc:232] RPC call timeout handler was delayed by -1.603557854000087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1575s for a response
I20250116 02:59:23.605010 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.606083 21335 raft_consensus.cc:2388] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:23.583356 22899 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20250116 02:59:23.574765 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:23.634896 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 247049us
I20250116 02:59:23.643898 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:23.582396 22874 raft_consensus.cc:695] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:23.641187 20676 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:23.634516 22865 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250116 02:59:23.643569 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 80218.3us
I20250116 02:59:23.652681 21302 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:23.637131 20571 leader_election.cc:400] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:23.654206 20571 leader_election.cc:403] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250116 02:59:23.635421 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.655652 21335 raft_consensus.cc:2388] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:23.656343 20723 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:23.484092 22878 consensus_queue.cc:237] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:23.665321 20720 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.652093 22865 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:23.583957 22895 consensus_queue.cc:1046] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20250116 02:59:23.666489 22865 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 1 FOLLOWER]: Advancing to term 2
W20250116 02:59:23.670082 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 86711.5us
I20250116 02:59:23.670984 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:23.656953 22891 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:23.432555 22518 raft_consensus.cc:2949] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:23.654808 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.682309 20647 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:23.682724 21302 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.663405 22766 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:23.687884 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:23.689141 20601 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:23.689220 20720 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.693253 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.669606 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.680892 22815 raft_consensus.cc:2743] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:23.704627 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.705709 20601 raft_consensus.cc:2371] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 17446c8dacf04461acb843547665250f in term 3. Re-sending same reply.
W20250116 02:59:23.697160 22882 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20250116 02:59:23.685858 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.644954 22873 consensus_queue.cc:237] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.711867 21302 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.703564 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.740178 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.741626 21302 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:23.713132 22612 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:23.742956 22612 raft_consensus.cc:695] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:23.743973 22612 consensus_queue.cc:237] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:23.712442 22904 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:23.686492 22887 consensus_queue.cc:1046] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20250116 02:59:23.740444 21330 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:23.690287 22825 consensus_queue.cc:1046] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250116 02:59:23.748621 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:23.742179 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:23.751670 20606 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
W20250116 02:59:23.752336 20647 leader_election.cc:336] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Remote error: Timed out: Call waited in the queue past client deadline
I20250116 02:59:23.709863 22755 raft_consensus.cc:2798] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:23.721350 21327 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:23.760179 21302 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:23.763948 20606 pending_rounds.cc:77] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f: Aborting all ops after (but not including) 5
I20250116 02:59:23.713310 22895 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:23.758440 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.712142 22900 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:23.764456 20606 pending_rounds.cc:99] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250116 02:59:23.765015 20606 raft_consensus.cc:2883] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:23.722244 22815 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:23.768025 21334 raft_consensus.cc:2363] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4.
I20250116 02:59:23.721401 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:23.766350 20647 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:23.772156 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 02:59:23.775760 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 119467us
I20250116 02:59:23.778692 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 344) took 1100 ms. Trace:
I20250116 02:59:23.779142 20606 rpcz_store.cc:276] 0116 02:59:22.678066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.267275 (+589209us) service_pool.cc:225] Handling call
0116 02:59:23.283667 (+ 16392us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:23.778362 (+494695us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:23.778577 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:23.767297 22755 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:23.783146 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:23.769965 22887 consensus_queue.cc:1225] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer 57fcfcd5ab01438c9a4baf8a61871aa6 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250116 02:59:23.770956 22884 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:23.773289 22904 raft_consensus.cc:695] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:23.777258 22660 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:23.775846 22867 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:23.795830 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:23.796617 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:23.793483 22902 log.cc:926] Time spent T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.104s user 0.000s sys 0.001s
I20250116 02:59:23.781868 22895 raft_consensus.cc:2758] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250116 02:59:23.740275 22909 consensus_queue.cc:1046] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20250116 02:59:23.811470 22909 raft_consensus.cc:3049] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:23.811887 22909 raft_consensus.cc:738] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:23.812976 22909 consensus_queue.cc:260] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.816684 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:23.817492 20679 raft_consensus.cc:3049] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:23.787699 22755 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:23.809080 22893 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:23.809509 22920 consensus_queue.cc:1046] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250116 02:59:23.791193 22900 consensus_queue.cc:1046] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250116 02:59:23.845821 20749 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:23.769428 22874 consensus_queue.cc:237] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:23.805675 20601 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:23.804798 20723 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:23.797920 22796 consensus_meta.cc:287] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.912s user 0.010s sys 0.003s
I20250116 02:59:23.794317 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:23.807078 22660 raft_consensus.cc:695] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:23.842298 20604 raft_consensus.cc:1235] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:23.814441 22909 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:23.872970 20603 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
W20250116 02:59:23.841233 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 152097us
W20250116 02:59:23.844267 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 91499.3us
I20250116 02:59:23.799870 22884 raft_consensus.cc:695] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:23.788465 22868 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
W20250116 02:59:23.838507 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 56962.5us
W20250116 02:59:23.841776 22894 log.cc:926] Time spent T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.272s user 0.001s sys 0.000s
I20250116 02:59:23.868794 20605 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250116 02:59:23.865788 20602 consensus_meta.cc:287] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.515s user 0.008s sys 0.004s
I20250116 02:59:23.817845 20679 raft_consensus.cc:738] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:23.809276 22898 log.cc:926] Time spent T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.200s user 0.000s sys 0.001s
I20250116 02:59:23.868328 20601 raft_consensus.cc:2949] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:23.816722 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:23.882103 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 77390.3us
I20250116 02:59:23.875416 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:23.883708 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:23.868240 22815 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:23.950690 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 62759.7us
W20250116 02:59:23.948611 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 61681.7us
W20250116 02:59:23.978928 22923 log.cc:926] Time spent T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20250116 02:59:23.999662 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1607) took 1041 ms. Trace:
I20250116 02:59:24.067536 20603 rpcz_store.cc:276] 0116 02:59:22.958726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.064130 (+105404us) service_pool.cc:225] Handling call
0116 02:59:23.999471 (+935341us) spinlock_profiling.cc:243] Waited 790 ms on lock 0x7b540010a254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:23.999590 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1817585024}
I20250116 02:59:23.867556 22901 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:23.845180 22792 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:24.038833 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 163516us
I20250116 02:59:24.038563 20723 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:23.871979 22530 consensus_queue.cc:1046] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20250116 02:59:23.886745 20602 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:23.934634 20604 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:24.019599 22846 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:24.019629 21330 raft_consensus.cc:2388] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:24.060345 22796 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:24.102496 20679 consensus_queue.cc:260] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:23.933208 22887 consensus_queue.cc:1046] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.137s
I20250116 02:59:24.108144 20677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:24.069764 22884 consensus_queue.cc:237] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.064939 20674 raft_consensus.cc:2371] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4. Re-sending same reply.
I20250116 02:59:24.001919 22924 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.174s
I20250116 02:59:24.125097 20677 raft_consensus.cc:604] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:24.125666 20677 raft_consensus.cc:612] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:23.983268 22920 consensus_queue.cc:1046] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.117s
I20250116 02:59:23.958897 22612 consensus_queue.cc:1046] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.142s
I20250116 02:59:24.126484 22815 raft_consensus.cc:3049] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:24.130867 22815 raft_consensus.cc:738] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:23.961338 22904 consensus_queue.cc:237] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.131827 22815 consensus_queue.cc:260] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.133374 22815 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:24.097780 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 59306.1us
I20250116 02:59:24.102939 22892 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:24.095890 20571 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:23.960937 22867 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.136s
W20250116 02:59:24.098110 22665 consensus_meta.cc:287] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.846s user 0.011s sys 0.003s
I20250116 02:59:24.147521 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1476) took 1037 ms. Trace:
I20250116 02:59:24.148746 20723 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:24.209043 20682 rpcz_store.cc:276] 0116 02:59:23.110578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.126947 (+ 16369us) service_pool.cc:225] Handling call
0116 02:59:24.147256 (+1020309us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b5400086bd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:24.147442 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467504896}
W20250116 02:59:24.217669 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 69284.8us
I20250116 02:59:24.223925 22899 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:24.229990 22796 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:24.230978 22919 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election won for term 3
W20250116 02:59:24.236663 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 140873us
I20250116 02:59:24.237661 20571 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:24.119837 22792 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:24.202113 22892 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:24.212340 22902 log.cc:926] Time spent T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20250116 02:59:24.251261 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 189796us
I20250116 02:59:24.271365 20571 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:24.214581 22924 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:22.800063 20753 rpcz_store.cc:276] 0116 02:59:20.951544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.725183 (+1773639us) service_pool.cc:225] Handling call
0116 02:59:22.759628 (+ 34445us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:22.796476 (+ 36848us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:22.796520 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:22.796574 (+ 54us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:22.796616 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:22.796690 (+ 74us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:22.796838 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8594}
I20250116 02:59:24.141465 22660 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.112799 22846 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.234545 22878 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:24.268986 22919 raft_consensus.cc:695] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:24.253491 22530 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:24.271065 22892 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:24.214259 22665 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:24.317297 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 46007.6us
I20250116 02:59:24.285985 22825 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:24.333283 20571 leader_election.cc:400] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4.
I20250116 02:59:24.339601 20571 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:24.341912 20647 leader_election.cc:336] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Remote error: Timed out: Call waited in the queue past client deadline
I20250116 02:59:24.346061 20647 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:24.354827 20681 raft_consensus.cc:2388] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:24.332291 22893 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:24.337900 22600 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:24.335917 22796 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:24.331727 22530 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.317622 22924 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.343849 22825 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:24.390427 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 144878us
I20250116 02:59:24.452553 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:24.443498 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:24.447027 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1479) took 1314 ms. Trace:
I20250116 02:59:24.524761 20676 rpcz_store.cc:276] 0116 02:59:23.132386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.133200 (+ 814us) service_pool.cc:225] Handling call
0116 02:59:23.172966 (+ 39766us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:23.653516 (+480550us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:24.446692 (+793176us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b5400324954. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:24.446917 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1076257024}
W20250116 02:59:24.526095 21299 leader_election.cc:336] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 1.901s (SENT)
I20250116 02:59:24.526710 21299 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:24.533563 21332 raft_consensus.cc:2363] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:24.376526 21330 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250116 02:59:24.475834 22518 consensus_meta.cc:287] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.702s user 0.011s sys 0.000s
W20250116 02:59:24.403324 22865 consensus_meta.cc:287] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.729s user 0.005s sys 0.008s
I20250116 02:59:24.465968 22801 raft_consensus.cc:3049] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:24.450637 22766 raft_consensus.cc:3049] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:24.367971 21327 raft_consensus.cc:1212] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Deduplicated request from leader. Original: 1.4->[3.5-3.5] Dedup: 3.5->[]
I20250116 02:59:24.408844 22792 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:24.542016 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 477) took 1055 ms. Trace:
I20250116 02:59:24.595997 21332 rpcz_store.cc:276] 0116 02:59:23.479320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.479541 (+ 221us) service_pool.cc:225] Handling call
0116 02:59:24.443302 (+963761us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:24.534306 (+ 91004us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8095104}
I20250116 02:59:24.438774 22934 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:24.450893 22715 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:24.541731 22909 consensus_meta.cc:287] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.661s user 0.010s sys 0.000s
I20250116 02:59:24.413878 22878 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:24.611326 21327 raft_consensus.cc:1235] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250116 02:59:24.550976 22801 raft_consensus.cc:738] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:24.616591 20676 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:24.378407 22600 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:24.551607 22753 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:24.591095 22766 raft_consensus.cc:738] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:24.409592 22925 log.cc:926] Time spent T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.280s user 0.000s sys 0.001s
I20250116 02:59:24.618470 22665 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:24.633344 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 414533us
I20250116 02:59:24.398541 22893 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:24.642251 20647 leader_election.cc:336] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.530s (SENT)
W20250116 02:59:24.645599 22939 log.cc:926] Time spent T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20250116 02:59:24.633179 22846 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:24.651029 22940 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:24.652312 20647 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:24.638810 22600 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:24.555243 22865 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.632560 22934 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:24.618299 22715 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.637392 22952 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:24.656117 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:24.682724 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 45314.2us
W20250116 02:59:24.683815 20647 leader_election.cc:336] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.607s (SENT)
I20250116 02:59:24.684049 22966 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:24.676196 22824 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:24.684602 22966 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.684875 20647 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:24.685494 22824 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:24.687672 22824 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:24.687896 22766 consensus_queue.cc:260] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:24.687906 22966 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:24.677906 22940 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.691413 22940 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:24.692993 22825 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:24.694175 22530 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:24.673516 22930 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250116 02:59:24.698814 22801 consensus_queue.cc:260] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.684739 22957 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:24.652309 22892 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.704118 22903 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:24.705135 22892 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:24.660277 22878 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:24.413308 22919 consensus_queue.cc:237] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:24.699963 22801 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:24.710558 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 54510.6us
I20250116 02:59:24.706024 21329 raft_consensus.cc:2949] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:24.720779 20571 leader_election.cc:400] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:24.721155 20571 leader_election.cc:403] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250116 02:59:24.679728 22665 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:24.689042 22766 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:24.712057 20674 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.008s sys 0.020s
I20250116 02:59:24.737278 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:24.720227 22824 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:24.740401 21336 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:24.742013 22924 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:24.743577 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:24.743745 21327 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:24.729343 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:24.741079 20674 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s user 0.009s sys 0.021s
I20250116 02:59:24.738041 22825 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:24.715651 22878 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.758710 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:24.676546 22934 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:24.679507 22952 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.048079 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:24.762568 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:24.761801 20604 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250116 02:59:25.067121 20606 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
W20250116 02:59:24.793684 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 116391us
I20250116 02:59:25.095448 20604 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250116 02:59:25.081024 20678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1472) took 2018 ms (client timeout 1767 ms). Trace:
W20250116 02:59:24.788075 22755 consensus_meta.cc:287] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.942s user 0.010s sys 0.003s
W20250116 02:59:25.048746 22955 log.cc:926] Time spent T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.316s user 0.000s sys 0.001s
I20250116 02:59:24.934947 22715 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:25.119115 20678 rpcz_store.cc:269] 0116 02:59:23.063156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.072767 (+ 9611us) service_pool.cc:225] Handling call
0116 02:59:24.762365 (+1689598us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:25.080905 (+318540us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4488576}
W20250116 02:59:25.049351 22958 log.cc:926] Time spent T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.305s user 0.001s sys 0.000s
I20250116 02:59:24.820514 22966 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:25.124531 20606 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:24.867090 20602 raft_consensus.cc:2388] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:25.126890 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 57422.6us
I20250116 02:59:24.925279 22825 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:25.047781 22748 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:25.046288 22967 log.cc:926] Time spent T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
W20250116 02:59:25.039496 22882 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20250116 02:59:24.945762 22899 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.190s
I20250116 02:59:24.750424 22824 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.001544 22960 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:24.780428 22965 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:24.973692 22874 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:25.206914 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 345260us
I20250116 02:59:25.002310 22846 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:25.244282 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 1.17537e+06us
W20250116 02:59:25.387943 20681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1535) took 1903 ms (client timeout 1841 ms). Trace:
I20250116 02:59:25.161572 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1945) took 2759 ms. Trace:
I20250116 02:59:25.521265 20608 rpcz_store.cc:276] 0116 02:59:22.407517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.905074 (+497557us) service_pool.cc:225] Handling call
0116 02:59:22.957849 (+ 52775us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:23.299538 (+341689us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:25.161320 (+1861782us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54000578d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:25.161502 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":784313600}
I20250116 02:59:25.156821 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:25.518431 20681 rpcz_store.cc:269] 0116 02:59:23.484197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.585221 (+101024us) service_pool.cc:225] Handling call
0116 02:59:25.387644 (+1802423us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b54001905d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:25.387821 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1767216512}
I20250116 02:59:25.154927 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1539) took 1576 ms. Trace:
I20250116 02:59:25.529616 20682 rpcz_store.cc:276] 0116 02:59:23.578159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:24.213911 (+635752us) service_pool.cc:225] Handling call
0116 02:59:24.253043 (+ 39132us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:24.361443 (+108400us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:24.361485 (+ 42us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:24.376792 (+ 15307us) log.cc:844] Serialized 26 byte log entry
0116 02:59:24.662758 (+285966us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:24.662806 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:24.662964 (+ 158us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:25.154590 (+491626us) raft_consensus.cc:1680] finished
0116 02:59:25.154642 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:25.154835 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":154555,"spinlock_wait_cycles":8322944,"thread_start_us":359584,"threads_started":2,"wal-append.queue_time_us":331417}
I20250116 02:59:25.532253 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:25.160238 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 331) took 2852 ms. Trace:
I20250116 02:59:25.628067 20607 rpcz_store.cc:276] 0116 02:59:22.307368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.856600 (+549232us) service_pool.cc:225] Handling call
0116 02:59:22.856890 (+ 290us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:23.077122 (+220232us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:25.159924 (+2082802us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b5400109d54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:25.160159 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":431360896}
W20250116 02:59:25.636806 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 120400us
W20250116 02:59:25.236414 22963 log.cc:926] Time spent T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
W20250116 02:59:25.241358 22962 log.cc:926] Time spent T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.274s user 0.001s sys 0.000s
I20250116 02:59:25.647177 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:25.636418 20649 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:25.242342 22905 log.cc:926] Time spent T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 1.593s user 0.004s sys 0.020s
W20250116 02:59:25.277243 22957 consensus_meta.cc:287] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.577s user 0.007s sys 0.005s
I20250116 02:59:25.657881 22957 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:25.658257 22943 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:25.160001 22969 log.cc:926] Time spent T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20250116 02:59:25.642865 22874 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.162117 22934 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:25.365037 22865 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:25.382747 22933 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.020s sys 0.011s
I20250116 02:59:25.655441 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:25.313337 22825 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.155242 22715 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:25.384127 22927 raft_consensus.cc:3049] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:25.656140 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:25.132244 22755 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.642273 22919 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:25.410079 22796 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:25.246345 22612 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:25.663674 22933 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.912s user 0.023s sys 0.022s
I20250116 02:59:25.527376 22947 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:25.378858 22952 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:25.162117 22920 raft_consensus.cc:3049] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Stepping down as leader of term 4
W20250116 02:59:25.381405 22890 thread.cc:635] apply [worker] (thread pool) Time spent creating pthread: real 1.923s user 0.021s sys 0.028s
I20250116 02:59:25.662199 22874 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:25.428181 22871 raft_consensus.cc:3049] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
W20250116 02:59:25.286108 22884 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.007s sys 0.021s
I20250116 02:59:25.655741 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:25.671886 22927 raft_consensus.cc:738] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:25.137497 22893 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:25.673652 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:25.313056 22960 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.658584 22957 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:25.234477 22878 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:25.677384 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:25.678011 20680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 336) took 2967 ms (client timeout 1901 ms). Trace:
W20250116 02:59:25.700888 22890 thread.cc:601] apply [worker] (thread pool) Time spent starting thread: real 2.372s user 0.021s sys 0.044s
W20250116 02:59:25.658871 21329 consensus_meta.cc:287] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.793s user 0.008s sys 0.003s
I20250116 02:59:25.651180 22660 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:25.684747 20681 pending_rounds.cc:77] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Aborting all ops after (but not including) 5
I20250116 02:59:25.715230 20681 pending_rounds.cc:99] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250116 02:59:25.710033 20752 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:25.715521 20681 raft_consensus.cc:2883] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:25.683229 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:25.717025 20757 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:25.660063 22974 log.cc:926] Time spent T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.220s user 0.000s sys 0.000s
I20250116 02:59:25.692551 20676 raft_consensus.cc:1235] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:25.692864 22947 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.698968 22920 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:25.682873 22949 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20250116 02:59:25.695139 22863 consensus_queue.cc:1046] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:25.674486 22934 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:25.702507 20682 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:25.664294 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1971) took 1428 ms. Trace:
I20250116 02:59:25.703665 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:25.742344 20753 raft_consensus.cc:2388] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 2: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:25.313258 22965 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:25.655014 20754 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.049s sys 0.133s
W20250116 02:59:25.710038 20680 rpcz_store.cc:269] 0116 02:59:22.710332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.748852 (+ 38520us) service_pool.cc:225] Handling call
0116 02:59:25.677768 (+2928916us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54001c66d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:25.677900 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":448516992}
I20250116 02:59:25.741532 20603 rpcz_store.cc:276] 0116 02:59:24.236047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:24.251682 (+ 15635us) service_pool.cc:225] Handling call
0116 02:59:24.291349 (+ 39667us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:24.334736 (+ 43387us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:24.334772 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:24.349045 (+ 14273us) log.cc:844] Serialized 26 byte log entry
0116 02:59:24.626780 (+277735us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:24.626833 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:24.626925 (+ 92us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:25.664050 (+1037125us) raft_consensus.cc:1680] finished
0116 02:59:25.664073 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:25.664225 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":46631,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":3816,"thread_start_us":259314,"threads_started":2,"wal-append.queue_time_us":309860}
W20250116 02:59:25.696820 22600 consensus_meta.cc:287] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 1.020s user 0.010s sys 0.000s
I20250116 02:59:25.713272 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:25.757452 20756 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:59:25.663445 22846 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:25.704355 22884 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.030s user 0.010s sys 0.026s
I20250116 02:59:25.693117 22612 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.700686 22952 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:25.702060 22871 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:25.696210 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:25.676609 22715 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.725427 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:25.725728 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 444186us
W20250116 02:59:25.735705 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 496181us
W20250116 02:59:25.734491 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 422271us
I20250116 02:59:25.658864 22943 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.711143 22821 raft_consensus.cc:3049] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:25.687119 22796 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:25.710650 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:25.744767 22965 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:25.748309 22863 consensus_queue.cc:1225] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer 57fcfcd5ab01438c9a4baf8a61871aa6 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250116 02:59:25.685667 22919 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.756672 20749 raft_consensus.cc:2363] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4.
I20250116 02:59:25.707571 22874 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20250116 02:59:25.720707 22957 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.737655 22665 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20250116 02:59:25.723179 22660 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:25.724421 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1606) took 2803 ms. Trace:
I20250116 02:59:25.779966 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:26.014977 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 328046us
I20250116 02:59:26.059123 22665 consensus_queue.cc:1046] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:59:26.060262 22518 consensus_queue.cc:1046] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:26.060708 22952 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:25.763267 22824 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.061618 22825 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:25.780797 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1973) took 1282 ms. Trace:
I20250116 02:59:25.815388 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:26.019866 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 332078us
I20250116 02:59:25.762463 20757 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:25.799584 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:25.749734 20754 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s user 0.049s sys 0.133s
I20250116 02:59:26.061764 22824 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:25.763855 22983 log.cc:926] Time spent T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250116 02:59:25.812459 22927 consensus_queue.cc:260] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:26.023778 22986 log.cc:926] Time spent T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.231s user 0.000s sys 0.001s
I20250116 02:59:26.062829 22952 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:25.894287 20647 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.539s (SENT)
I20250116 02:59:25.740816 22934 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:26.065476 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:26.064178 20571 leader_election.cc:336] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Timed out: RequestConsensusVote RPC to 127.18.238.132:33959 timed out after 1.898s (SENT)
I20250116 02:59:26.122925 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:26.065287 20601 rpcz_store.cc:276] 0116 02:59:24.497894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:24.508806 (+ 10912us) service_pool.cc:225] Handling call
0116 02:59:24.509176 (+ 370us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:24.509533 (+ 357us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:24.509558 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:24.510111 (+ 553us) log.cc:844] Serialized 26 byte log entry
0116 02:59:24.767639 (+257528us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:24.767705 (+ 66us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:24.767932 (+ 227us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:25.780532 (+1012600us) raft_consensus.cc:1680] finished
0116 02:59:25.780560 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:25.780723 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":19798,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":53452,"spinlock_wait_cycles":10112,"thread_start_us":257108,"threads_started":1,"wal-append.queue_time_us":860542}
I20250116 02:59:25.844113 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.075721 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:25.822322 22755 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.060336 22821 raft_consensus.cc:738] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:26.076278 20755 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:26.122356 20752 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:26.128212 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.129004 20752 raft_consensus.cc:2388] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:26.063725 22612 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.130779 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.131523 21336 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:26.132544 20750 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:26.134328 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.135025 20748 raft_consensus.cc:2363] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4.
I20250116 02:59:26.136132 21334 raft_consensus.cc:1235] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:26.136565 20750 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:26.136158 20756 raft_consensus.cc:1235] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250116 02:59:26.144796 21331 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250116 02:59:26.128448 22801 consensus_meta.cc:287] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.411s user 0.013s sys 0.001s
I20250116 02:59:26.065485 22846 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:25.971431 22949 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:26.071449 22960 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.159272 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.189869 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.153417 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 343) took 2436 ms. Trace:
I20250116 02:59:26.126449 22934 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:26.126259 22927 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:26.190672 20752 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:26.192327 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.192938 20752 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:26.193677 20757 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:26.189669 21334 raft_consensus.cc:1235] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:26.146588 21330 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250116 02:59:26.148483 22957 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.174703 22952 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:26.227731 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.190773 20678 rpcz_store.cc:276] 0116 02:59:23.716770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:25.692540 (+1975770us) service_pool.cc:225] Handling call
0116 02:59:25.705693 (+ 13153us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:25.705963 (+ 270us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:25.705991 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:25.706467 (+ 476us) log.cc:844] Serialized 26 byte log entry
0116 02:59:25.982508 (+276041us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:25.982552 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:25.982663 (+ 111us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:26.153139 (+170476us) raft_consensus.cc:1680] finished
0116 02:59:26.153165 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:26.153340 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":233785,"threads_started":1,"wal-append.queue_time_us":405010}
I20250116 02:59:26.242847 22715 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.243024 22952 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.240350 20675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1464) took 3244 ms. Trace:
I20250116 02:59:26.245229 22952 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.244756 20675 rpcz_store.cc:276] 0116 02:59:22.995896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:22.997941 (+ 2045us) service_pool.cc:225] Handling call
0116 02:59:23.041409 (+ 43468us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:23.090914 (+ 49505us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:23.095024 (+ 4110us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:23.168009 (+ 72985us) log.cc:844] Serialized 26 byte log entry
0116 02:59:23.495316 (+327307us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:23.498577 (+ 3261us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:23.550541 (+ 51964us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:26.158309 (+2607768us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b2400107f48. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323c2d9d4 00007fc323c2badb 00007fc323cba289 00007fc323cb9f15 00007fc323cb9470 00007fc3269b9f88 00007fc3269ba6e8 00007fc326991f98 00007fc32698d3aa 00007fc32698c7f6
0116 02:59:26.159587 (+ 1278us) raft_consensus.cc:1680] finished
0116 02:59:26.159610 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:26.234238 (+ 74628us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13148,"raft.queue_time_us":33068,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":14582,"spinlock_wait_cycles":27831680,"thread_start_us":204592,"threads_started":1,"wal-append.queue_time_us":445076}
I20250116 02:59:26.247530 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:26.250388 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:26.241678 22796 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:26.197643 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.147878 22987 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:26.151867 22995 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:26.234992 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.196846 22998 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:26.252446 22998 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.253756 20682 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:26.214720 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.252447 21334 raft_consensus.cc:2388] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 2: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:26.147627 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 396) took 1494 ms. Trace:
I20250116 02:59:26.240865 22873 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:26.251056 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:26.230844 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 336591us
I20250116 02:59:26.142202 22825 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:26.196033 22949 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:26.299989 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:26.090523 22824 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:26.196221 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 132095us
I20250116 02:59:26.225381 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:26.312168 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 52213.7us
W20250116 02:59:26.265024 20572 connection.cc:232] RPC call timeout handler was delayed by -1.9641198379999878s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1789s for a response
W20250116 02:59:26.312978 20572 connection.cc:232] RPC call timeout handler was delayed by -1.9173198380000258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
W20250116 02:59:26.313324 20572 connection.cc:232] RPC call timeout handler was delayed by -1.8777198379999618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1885s for a response
W20250116 02:59:26.313622 20572 connection.cc:232] RPC call timeout handler was delayed by -1.8498198380000304s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1916s for a response
W20250116 02:59:26.314112 20572 connection.cc:232] RPC call timeout handler was delayed by -1.817419837999978s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
W20250116 02:59:26.314491 20572 connection.cc:232] RPC call timeout handler was delayed by -1.7994198379999489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19720000000000001s for a response
W20250116 02:59:26.314819 20572 connection.cc:232] RPC call timeout handler was delayed by -1.7778198380000276s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1996s for a response
W20250116 02:59:26.315107 20572 connection.cc:232] RPC call timeout handler was delayed by -1.7697198380000145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2005s for a response
W20250116 02:59:26.315384 20572 connection.cc:232] RPC call timeout handler was delayed by -1.7283630029999131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
W20250116 02:59:26.315662 20572 connection.cc:232] RPC call timeout handler was delayed by -1.721119837999936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20590000000000003s for a response
W20250116 02:59:26.315958 20572 connection.cc:232] RPC call timeout handler was delayed by -1.6869198379999943s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
W20250116 02:59:26.316254 20572 connection.cc:232] RPC call timeout handler was delayed by -1.6779198379999798s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
W20250116 02:59:26.317427 20572 connection.cc:232] RPC call timeout handler was delayed by -1.6716198379999696s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2114s for a response
W20250116 02:59:26.317761 20572 connection.cc:232] RPC call timeout handler was delayed by -1.5960198379999611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
W20250116 02:59:26.318053 20572 connection.cc:232] RPC call timeout handler was delayed by -1.5717198379999218s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
W20250116 02:59:26.318446 20572 connection.cc:232] RPC call timeout handler was delayed by -1.5006198379999205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2304s for a response
W20250116 02:59:26.318863 20572 connection.cc:232] RPC call timeout handler was delayed by -1.2936198379999269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2534s for a response
W20250116 02:59:26.319159 20572 connection.cc:232] RPC call timeout handler was delayed by -1.2302012449999893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1575s for a response
W20250116 02:59:26.319435 20572 connection.cc:232] RPC call timeout handler was delayed by -1.0592012450000539s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1765s for a response
W20250116 02:59:26.319713 20572 connection.cc:232] RPC call timeout handler was delayed by -1.0204806779998989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19590000000000002s for a response
I20250116 02:59:26.323746 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:25.765322 22600 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.251586 22891 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:26.301970 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.299657 22715 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:25.812261 22920 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.306233 20646 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:26.299089 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.307292 22936 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20250116 02:59:26.307580 22753 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:26.261348 22987 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.300670 20675 raft_consensus.cc:2388] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:26.151177 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.301172 20678 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:26.327973 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.341874 20757 raft_consensus.cc:2388] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:26.342942 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:26.300001 22873 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:26.198235 23002 log.cc:926] Time spent T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20250116 02:59:26.331537 20604 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:26.371356 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.374414 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:26.381907 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:26.309805 22748 consensus_meta.cc:287] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.999s user 0.010s sys 0.002s
I20250116 02:59:26.337992 22753 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.391606 21335 raft_consensus.cc:2405] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250116 02:59:26.392637 22753 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:26.394135 22924 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:26.337531 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:26.395526 20603 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 1.
I20250116 02:59:26.338416 20601 raft_consensus.cc:1235] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:26.327910 22796 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:26.383006 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.335770 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:26.401818 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 66127.1us
I20250116 02:59:26.385891 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:26.401851 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.405364 20755 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:26.398252 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:26.403661 20604 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:26.067142 20599 rpcz_store.cc:276] 0116 02:59:22.918830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.275435 (+356605us) service_pool.cc:225] Handling call
0116 02:59:23.275895 (+ 460us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:23.390675 (+114780us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:23.419179 (+ 28504us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:23.432909 (+ 13730us) log.cc:844] Serialized 26 byte log entry
0116 02:59:23.555052 (+122143us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:23.561580 (+ 6528us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:23.571777 (+ 10197us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:25.241595 (+1669818us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:25.697600 (+456005us) raft_consensus.cc:1680] finished
0116 02:59:25.697634 (+ 34us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:25.721781 (+ 24147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6286080,"thread_start_us":62510,"threads_started":1,"wal-append.queue_time_us":154551}
I20250116 02:59:26.368266 22880 raft_consensus.cc:3049] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:26.402972 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:26.204380 22821 consensus_queue.cc:260] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.381603 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:26.340193 22887 consensus_queue.cc:1046] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20250116 02:59:26.344589 22999 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20250116 02:59:26.416210 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:26.354067 20748 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:26.337059 22932 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20250116 02:59:26.394870 22924 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.334537 22715 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.348798 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.426985 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.411509 20604 raft_consensus.cc:1235] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:26.379632 22920 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:26.348212 22868 consensus_queue.cc:1046] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20250116 02:59:26.425462 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:26.410907 22957 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:26.333884 22825 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:26.434418 20603 raft_consensus.cc:2371] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 17446c8dacf04461acb843547665250f in term 4. Re-sending same reply.
I20250116 02:59:26.439862 20757 raft_consensus.cc:2388] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:26.444329 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 141438us
I20250116 02:59:26.450167 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.457218 20755 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
W20250116 02:59:26.451922 20720 connection.cc:232] RPC call timeout handler was delayed by -2.5905429199999617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1505s for a response
W20250116 02:59:26.234992 22766 consensus_meta.cc:287] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 1.501s user 0.015s sys 0.003s
W20250116 02:59:26.458189 20720 connection.cc:232] RPC call timeout handler was delayed by -2.5023429200000464s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
I20250116 02:59:26.417212 22880 raft_consensus.cc:738] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:26.459635 21300 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:26.432655 22846 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:26.428659 22943 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.438634 20602 raft_consensus.cc:1270] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:26.460976 20606 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:26.467691 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:26.473769 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 342) took 2758 ms. Trace:
I20250116 02:59:26.474582 20601 raft_consensus.cc:2388] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:26.429651 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:26.472064 20608 raft_consensus.cc:1270] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250116 02:59:26.486040 20608 pending_rounds.cc:77] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Aborting all ops after (but not including) 1
I20250116 02:59:26.468533 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:26.486251 20603 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:26.486577 20608 pending_rounds.cc:99] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250116 02:59:26.486937 20608 raft_consensus.cc:2883] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250116 02:59:26.461144 20720 connection.cc:232] RPC call timeout handler was delayed by -2.4285429200000408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1685s for a response
I20250116 02:59:26.332000 22891 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.413946 22796 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:26.474282 20681 rpcz_store.cc:276] 0116 02:59:23.715383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:25.676194 (+1960811us) service_pool.cc:225] Handling call
0116 02:59:25.684557 (+ 8363us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:25.716135 (+ 31578us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:25.716178 (+ 43us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:25.716670 (+ 492us) log.cc:844] Serialized 26 byte log entry
0116 02:59:26.069675 (+353005us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:26.069711 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:26.069780 (+ 69us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:26.473554 (+403774us) raft_consensus.cc:1680] finished
0116 02:59:26.473578 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:26.473702 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":204292,"threads_started":1,"wal-append.queue_time_us":318677}
I20250116 02:59:26.441926 22660 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:26.449615 22919 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.489153 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.509219 20755 raft_consensus.cc:2405] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250116 02:59:26.453266 22957 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:26.493681 20757 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:59:26.487504 20720 connection.cc:232] RPC call timeout handler was delayed by -2.4114429200000131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
W20250116 02:59:26.509961 20720 connection.cc:232] RPC call timeout handler was delayed by -2.3727429199999506s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20250116 02:59:26.510317 20720 connection.cc:232] RPC call timeout handler was delayed by -2.3547429200000352s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
I20250116 02:59:26.432160 22530 consensus_queue.cc:1046] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20250116 02:59:26.460242 22612 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:26.510588 20720 connection.cc:232] RPC call timeout handler was delayed by -2.2665429200000062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1865s for a response
W20250116 02:59:26.520949 20720 connection.cc:232] RPC call timeout handler was delayed by -2.2287429199999451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20250116 02:59:26.521299 20720 connection.cc:232] RPC call timeout handler was delayed by -2.183742919999986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
W20250116 02:59:26.521545 20720 connection.cc:232] RPC call timeout handler was delayed by -2.042442919999985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2114s for a response
W20250116 02:59:26.521824 20720 connection.cc:232] RPC call timeout handler was delayed by -1.9965429200000244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21650000000000003s for a response
I20250116 02:59:26.474052 22766 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:26.522307 20720 connection.cc:232] RPC call timeout handler was delayed by -1.9605429199999662s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22050000000000003s for a response
W20250116 02:59:26.522678 20720 connection.cc:232] RPC call timeout handler was delayed by -1.821042919999968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.236s for a response
I20250116 02:59:26.470531 22846 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:26.505115 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 125098us
I20250116 02:59:26.521883 20752 raft_consensus.cc:2363] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4.
I20250116 02:59:26.515542 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:26.529354 20601 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
W20250116 02:59:26.523053 20720 connection.cc:232] RPC call timeout handler was delayed by -1.6194429199999831s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2584s for a response
W20250116 02:59:26.530390 20720 connection.cc:232] RPC call timeout handler was delayed by -1.5114429200000359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27040000000000003s for a response
W20250116 02:59:26.530908 20720 connection.cc:232] RPC call timeout handler was delayed by -1.2450429199999462s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30000000000000004s for a response
W20250116 02:59:26.531303 20720 connection.cc:232] RPC call timeout handler was delayed by -1.1829429199999595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3069s for a response
W20250116 02:59:26.531790 20720 connection.cc:232] RPC call timeout handler was delayed by -1.0542429199999788s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32120000000000004s for a response
I20250116 02:59:26.491753 22801 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:26.473747 22994 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:26.549082 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:26.517850 22957 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:26.257651 20754 rpcz_store.cc:276] 0116 02:59:24.652751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:24.658584 (+ 5833us) service_pool.cc:225] Handling call
0116 02:59:24.668768 (+ 10184us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:24.694729 (+ 25961us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:24.894698 (+199969us) log.cc:844] Serialized 14 byte log entry
0116 02:59:26.085946 (+1191248us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:26.125916 (+ 39970us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:26.128142 (+ 2226us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:26.140373 (+ 12231us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:26.147525 (+ 7152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":438430,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":69,"raft.queue_time_us":67802,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":32708,"thread_start_us":1136636,"threads_started":2,"wal-append.queue_time_us":836978}
I20250116 02:59:26.544086 22952 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:26.530337 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:26.545542 22846 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:26.744136 20603 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:26.747869 20757 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:26.757510 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:26.617053 22994 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:26.816105 20681 raft_consensus.cc:2388] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:26.658798 20720 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:26.549628 22897 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250116 02:59:26.712419 22996 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.456s user 0.001s sys 0.006s
W20250116 02:59:26.771412 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 346012us
W20250116 02:59:26.756137 23010 log.cc:926] Time spent T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20250116 02:59:26.845465 22919 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:26.876952 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 340) took 3165 ms. Trace:
I20250116 02:59:26.877533 20676 rpcz_store.cc:276] 0116 02:59:23.711755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:24.636729 (+924974us) service_pool.cc:225] Handling call
0116 02:59:24.658412 (+ 21683us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:25.719594 (+1061182us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:26.876591 (+1156997us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001919d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:26.876860 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2377710848}
I20250116 02:59:26.883927 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:26.885542 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:26.755534 22949 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:26.887935 22753 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:26.759488 22796 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:26.758419 22715 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.655828 20599 raft_consensus.cc:1270] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:26.887935 22891 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.888487 22753 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:26.889633 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:26.890906 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:26.891407 22753 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:26.891618 23004 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.232s user 0.000s sys 0.000s
I20250116 02:59:26.634446 22824 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.891513 20599 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:26.869627 23005 log.cc:926] Time spent T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.400s user 0.000s sys 0.001s
W20250116 02:59:26.906971 20572 leader_election.cc:336] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.516s (SENT)
I20250116 02:59:27.032258 20572 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:26.866206 20571 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:26.928534 20748 raft_consensus.cc:2388] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:27.041826 22843 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:26.622447 22801 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:27.041486 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:27.043601 21332 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:27.044508 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.930908 22987 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:26.821200 22914 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:27.045436 22914 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:26.892647 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1972) took 2366 ms. Trace:
I20250116 02:59:27.047569 22879 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:26.931918 22825 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:26.932811 22873 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:26.934085 23008 log.cc:926] Time spent T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.237s user 0.000s sys 0.000s
W20250116 02:59:26.940681 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 76108.5us
I20250116 02:59:26.655159 22600 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:26.698379 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.615758 22612 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:27.040549 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 133636us
W20250116 02:59:27.041272 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 175209us
I20250116 02:59:26.867165 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.044580 22801 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:26.634861 22880 consensus_queue.cc:260] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.071460 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.080577 20571 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:27.081013 20676 raft_consensus.cc:2388] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:27.068667 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 344) took 3302 ms. Trace:
I20250116 02:59:27.084295 20680 rpcz_store.cc:276] 0116 02:59:23.766530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:25.750164 (+1983634us) service_pool.cc:225] Handling call
0116 02:59:25.750540 (+ 376us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:25.750806 (+ 266us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:25.750834 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:26.075261 (+324427us) log.cc:844] Serialized 26 byte log entry
0116 02:59:26.349480 (+274219us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:26.349517 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:26.349580 (+ 63us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:27.068189 (+718609us) raft_consensus.cc:1680] finished
0116 02:59:27.068330 (+ 141us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.068539 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":214503,"threads_started":1,"wal-append.queue_time_us":361749}
I20250116 02:59:27.085989 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:27.088129 20572 leader_election.cc:336] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.575s (SENT)
I20250116 02:59:27.046818 20600 rpcz_store.cc:276] 0116 02:59:24.497569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:24.537159 (+ 39590us) service_pool.cc:225] Handling call
0116 02:59:26.250174 (+1713015us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:26.250507 (+ 333us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:26.250799 (+ 292us) raft_consensus.cc:1529] Early marking committed up to index 2
0116 02:59:26.470222 (+219423us) log.cc:844] Serialized 14 byte log entry
0116 02:59:26.697424 (+227202us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:26.713106 (+ 15682us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:26.761035 (+ 47929us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:26.777232 (+ 16197us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:26.864342 (+ 87110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4391,"prepare.queue_time_us":383592,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":76548,"raft.run_wall_time_us":45154,"spinlock_wait_cycles":62275072,"thread_start_us":324271,"threads_started":2,"wal-append.queue_time_us":460304}
I20250116 02:59:27.095635 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:26.824209 22994 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:27.107512 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.108567 20680 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:27.047799 22914 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:27.114096 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:27.109532 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.105836 22715 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:27.112471 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 579996us
I20250116 02:59:27.127651 20603 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:27.131451 20603 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250116 02:59:27.139711 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 400) took 2444 ms. Trace:
I20250116 02:59:26.847066 22952 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:27.112680 20572 leader_election.cc:336] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.765s (SENT)
I20250116 02:59:27.046394 22987 consensus_queue.cc:1046] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20250116 02:59:27.148502 20571 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:27.351694 22715 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250116 02:59:27.352195 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 200383us
I20250116 02:59:27.352509 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 346) took 3474 ms. Trace:
I20250116 02:59:27.352548 22715 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:27.353032 20605 rpcz_store.cc:276] 0116 02:59:23.878415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.252544 (+2374129us) service_pool.cc:225] Handling call
0116 02:59:26.252937 (+ 393us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:26.253320 (+ 383us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:26.253351 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:26.253869 (+ 518us) log.cc:844] Serialized 26 byte log entry
0116 02:59:26.440535 (+186666us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:26.440588 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:26.440673 (+ 85us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:27.352248 (+911575us) raft_consensus.cc:1680] finished
0116 02:59:27.352275 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.352440 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1724,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":7226,"thread_start_us":186183,"threads_started":1,"wal-append.queue_time_us":227786}
I20250116 02:59:27.167106 20599 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:27.356287 20572 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:27.164592 22927 consensus_meta.cc:287] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.936s user 0.012s sys 0.000s
I20250116 02:59:27.357193 22600 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:27.152243 20720 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.148670 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:27.358249 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 245588us
I20250116 02:59:27.358544 22824 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:27.121551 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:27.359747 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 207616us
I20250116 02:59:27.360569 20720 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:27.360932 21335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 487) took 3654 ms (client timeout 1898 ms). Trace:
I20250116 02:59:27.361223 21327 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:27.156692 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:27.163323 22957 consensus_meta.cc:287] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.508s user 0.010s sys 0.000s
I20250116 02:59:27.363076 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:27.364655 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:27.365347 21332 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:27.169073 22920 consensus_meta.cc:287] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.710s user 0.009s sys 0.001s
I20250116 02:59:27.367889 20753 raft_consensus.cc:2405] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250116 02:59:27.367973 22949 consensus_queue.cc:1046] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:27.369632 22932 raft_consensus.cc:3049] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:27.370091 22932 raft_consensus.cc:738] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:27.148801 22843 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:27.371109 22932 consensus_queue.cc:260] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:27.142598 20751 rpcz_store.cc:276] 0116 02:59:24.695553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:24.711927 (+ 16374us) service_pool.cc:225] Handling call
0116 02:59:26.248066 (+1536139us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b58001f0598. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:26.248285 (+ 219us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:26.437331 (+189046us) raft_consensus.cc:1529] Early marking committed up to index 5
0116 02:59:26.661455 (+224124us) log.cc:844] Serialized 14 byte log entry
0116 02:59:27.139293 (+477838us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.139361 (+ 68us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:27.139433 (+ 72us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.139510 (+ 77us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.139634 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":431750,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":4086,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":213,"spinlock_wait_cycles":2583424,"thread_start_us":771292,"threads_started":3}
I20250116 02:59:27.382030 22957 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.383495 20601 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:27.385603 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 411) took 1241 ms. Trace:
I20250116 02:59:27.152098 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:27.318468 20633 heartbeater.cc:646] Failed to heartbeat to 127.18.238.190:37551 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.18.238.190:37551 timed out after 15.000s (SENT)
I20250116 02:59:27.407317 22957 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.408366 22897 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:27.357550 22987 raft_consensus.cc:3049] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:27.381215 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48834 (request call id 20) took 3825 ms. Trace:
I20250116 02:59:27.409008 22987 raft_consensus.cc:738] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:27.399956 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:27.121842 22879 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.409660 20677 rpcz_store.cc:276] 0116 02:59:23.556042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:24.107976 (+551934us) service_pool.cc:225] Handling call
0116 02:59:27.380937 (+3272961us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b54001c52d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d09bc3 00007fc32bc7082a 00007fc325349d14 00007fc325349ca2 00007fc325349bd1 00007fc325349b4c 00007fc325348e02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:27.381098 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37273984}
I20250116 02:59:27.404222 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 410) took 1260 ms. Trace:
I20250116 02:59:27.363332 22952 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.260777 22846 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.420575 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.412689 21330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 760) took 1108 ms. Trace:
W20250116 02:59:27.359009 20572 leader_election.cc:336] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.789s (SENT)
W20250116 02:59:27.408104 21335 rpcz_store.cc:269] 0116 02:59:23.706677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.710728 (+ 4051us) service_pool.cc:225] Handling call
0116 02:59:27.360679 (+3649951us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b540058f054. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:27.360833 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6094786432}
W20250116 02:59:27.351548 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 203135us
W20250116 02:59:27.167717 22934 consensus_meta.cc:287] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.856s user 0.008s sys 0.003s
I20250116 02:59:27.413175 22879 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:27.415231 22927 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:27.416070 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 407) took 1272 ms. Trace:
I20250116 02:59:27.430192 20757 rpcz_store.cc:276] 0116 02:59:26.143406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.776059 (+632653us) service_pool.cc:225] Handling call
0116 02:59:26.776400 (+ 341us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:26.776621 (+ 221us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:27.352272 (+575651us) log.cc:844] Serialized 14 byte log entry
0116 02:59:27.415670 (+ 63398us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.415730 (+ 60us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:27.415791 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.415865 (+ 74us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.416004 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":92761,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":1425,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":1109,"thread_start_us":429330,"threads_started":2,"wal-append.queue_time_us":63224}
I20250116 02:59:27.404771 20751 rpcz_store.cc:276] 0116 02:59:26.144372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.384473 (+1240101us) service_pool.cc:225] Handling call
0116 02:59:27.384950 (+ 477us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.385195 (+ 245us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.385224 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.385273 (+ 49us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:27.385316 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.385370 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.385527 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.410589 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 401) took 1331 ms. Trace:
I20250116 02:59:27.417335 20755 rpcz_store.cc:276] 0116 02:59:26.144133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.359850 (+1215717us) service_pool.cc:225] Handling call
0116 02:59:27.360287 (+ 437us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.360481 (+ 194us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.360507 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.360555 (+ 48us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:27.360595 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.360646 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.404095 (+ 43449us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:27.420326 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.011s sys 0.011s
W20250116 02:59:27.434934 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.011s sys 0.011s
W20250116 02:59:27.436523 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 57657us
I20250116 02:59:27.353308 22715 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:27.423462 20572 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:27.414676 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 54200.9us
I20250116 02:59:27.413671 22987 consensus_queue.cc:260] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.439308 20750 rpcz_store.cc:276] 0116 02:59:26.078905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.133465 (+ 54560us) service_pool.cc:225] Handling call
0116 02:59:26.136227 (+ 2762us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:26.136990 (+ 763us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.410372 (+1273382us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b5400370fd0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:27.410518 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5239168}
W20250116 02:59:27.424320 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 67463.9us
W20250116 02:59:27.414867 22918 log.cc:926] Time spent T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 3.662s user 0.021s sys 0.049s
I20250116 02:59:27.426090 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:27.428406 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:27.438630 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 79630.4us
I20250116 02:59:27.421589 22897 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:27.438913 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.409968 22994 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.446877 22612 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.442497 22748 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20250116 02:59:27.447755 22846 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.441730 20720 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.442507 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250116 02:59:27.422397 21330 rpcz_store.cc:276] 0116 02:59:26.304386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.411680 (+1107294us) service_pool.cc:225] Handling call
0116 02:59:27.412086 (+ 406us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.412311 (+ 225us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.412348 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.412395 (+ 47us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:27.412439 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.412505 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.412624 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.445967 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 413) took 1186 ms. Trace:
I20250116 02:59:27.445574 20676 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:27.450441 20751 rpcz_store.cc:276] 0116 02:59:26.259059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.439244 (+1180185us) service_pool.cc:225] Handling call
0116 02:59:27.445265 (+ 6021us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.445462 (+ 197us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.445487 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.445530 (+ 43us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:27.445594 (+ 64us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.445647 (+ 53us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.445877 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.454245 20750 pending_rounds.cc:77] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 4
I20250116 02:59:27.454631 20750 pending_rounds.cc:99] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250116 02:59:27.448114 20755 raft_consensus.cc:1235] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:27.454885 20750 raft_consensus.cc:2883] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:27.429646 22934 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.449515 22943 consensus_queue.cc:1046] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20250116 02:59:27.461354 22934 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.438409 22952 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:27.466125 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.477337 20677 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:27.477886 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1495) took 1059 ms. Trace:
I20250116 02:59:27.478219 20677 rpcz_store.cc:276] 0116 02:59:26.418852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.465743 (+1046891us) service_pool.cc:225] Handling call
0116 02:59:27.477815 (+ 12072us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.470955 20571 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:27.448235 22994 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:27.449965 22957 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:27.487279 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:27.488122 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 44050.2us
I20250116 02:59:27.490700 20720 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:27.491901 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:27.492152 20572 leader_election.cc:336] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.885s (SENT)
I20250116 02:59:27.492689 20572 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:27.492823 20676 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:27.462436 22987 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:27.453123 22612 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:27.484645 22846 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:27.503051 21332 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
W20250116 02:59:27.504622 20572 leader_election.cc:336] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.916s (SENT)
I20250116 02:59:27.504518 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:27.505059 20572 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:27.492502 22927 raft_consensus.cc:3054] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:27.514286 20676 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:27.491427 22957 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:27.494879 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 350) took 1078 ms. Trace:
I20250116 02:59:27.496941 22920 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:27.496052 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:27.502216 22952 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:27.523442 20572 leader_election.cc:336] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.972s (SENT)
I20250116 02:59:27.507022 22846 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:27.524255 20572 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:27.521667 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 71489.6us
W20250116 02:59:27.524205 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 48220.6us
W20250116 02:59:27.525171 20647 connection.cc:232] RPC call timeout handler was delayed by -1.2306821109999646s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2157s for a response
W20250116 02:59:27.525846 20647 connection.cc:232] RPC call timeout handler was delayed by -1.2126821109999355s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2177s for a response
I20250116 02:59:27.522441 21332 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250116 02:59:27.526599 20647 leader_election.cc:336] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.591s (SENT)
I20250116 02:59:27.527868 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:27.528018 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 449) took 1120 ms. Trace:
I20250116 02:59:27.520264 22957 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:27.529186 20647 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:27.529434 21332 rpcz_store.cc:276] 0116 02:59:26.407449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.506488 (+1099039us) service_pool.cc:225] Handling call
0116 02:59:27.521607 (+ 15119us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:27.527724 (+ 6117us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.527965 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.520882 22600 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:27.531800 20720 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:27.524446 22943 raft_consensus.cc:3049] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Stepping down as leader of term 2
W20250116 02:59:27.533532 20647 leader_election.cc:336] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.641s (SENT)
I20250116 02:59:27.533775 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1490) took 1144 ms. Trace:
W20250116 02:59:27.535149 20572 leader_election.cc:336] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.048s (SENT)
I20250116 02:59:27.531550 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:27.515540 22612 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.535888 20572 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:27.526859 22846 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:27.537266 20720 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.525648 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 406) took 1382 ms. Trace:
I20250116 02:59:27.537720 20676 rpcz_store.cc:276] 0116 02:59:26.389769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.527234 (+1137465us) service_pool.cc:225] Handling call
0116 02:59:27.533723 (+ 6489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.538791 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.539494 20676 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:27.542656 23032 consensus_queue.cc:1046] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20250116 02:59:27.545039 22612 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.548110 22914 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:27.550385 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 349) took 1141 ms. Trace:
I20250116 02:59:27.519696 20602 rpcz_store.cc:276] 0116 02:59:26.409912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.550847 (+140935us) service_pool.cc:225] Handling call
0116 02:59:26.784650 (+233803us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:26.928750 (+144100us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:26.928791 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:26.929454 (+ 663us) log.cc:844] Serialized 26 byte log entry
0116 02:59:27.123495 (+194041us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:27.123550 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.123648 (+ 98us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:27.475343 (+351695us) raft_consensus.cc:1680] finished
0116 02:59:27.475373 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.488481 (+ 13108us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1505,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"thread_start_us":193517,"threads_started":1,"wal-append.queue_time_us":499937}
W20250116 02:59:27.534662 23028 log.cc:926] Time spent T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20250116 02:59:27.528753 21331 raft_consensus.cc:1235] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250116 02:59:27.552484 20601 rpcz_store.cc:276] 0116 02:59:26.409516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.530099 (+120583us) service_pool.cc:225] Handling call
0116 02:59:26.622428 (+ 92329us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.384001 (+761573us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.550146 (+166145us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b5400108954. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:27.550325 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1749723008}
W20250116 02:59:27.558094 20572 leader_election.cc:336] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.059s (SENT)
I20250116 02:59:27.532887 22952 raft_consensus.cc:2743] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:27.535126 22920 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:27.561908 20572 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:27.563588 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:27.564538 21299 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:27.535187 22943 raft_consensus.cc:738] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:27.565524 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.566651 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1497) took 1109 ms. Trace:
I20250116 02:59:27.557322 22612 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:27.568914 20676 rpcz_store.cc:276] 0116 02:59:26.457784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.540087 (+1082303us) service_pool.cc:225] Handling call
0116 02:59:27.566557 (+ 26470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.570554 20720 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.558658 22600 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:27.575387 21335 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:27.578994 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:27.567638 22994 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:27.567775 22825 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:27.572410 22766 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:27.580449 22994 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:27.586804 20572 leader_election.cc:336] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.097s (SENT)
I20250116 02:59:27.588318 22943 consensus_queue.cc:260] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:27.591050 22825 raft_consensus.cc:695] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:27.602012 21330 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:27.593468 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:27.603040 20678 raft_consensus.cc:2388] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:27.603507 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1488) took 1254 ms. Trace:
I20250116 02:59:27.603818 20678 rpcz_store.cc:276] 0116 02:59:26.348956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.593090 (+1244134us) service_pool.cc:225] Handling call
0116 02:59:27.603434 (+ 10344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.604833 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:27.605439 22964 log.cc:926] Time spent T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 2.575s user 0.009s sys 0.007s
I20250116 02:59:27.598273 22994 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:27.601516 22825 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.577728 22920 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:27.609934 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 48460.4us
I20250116 02:59:27.610736 20720 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:27.612089 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1538) took 4038 ms. Trace:
I20250116 02:59:27.612157 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:27.614029 21334 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:27.614599 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 497) took 1418 ms. Trace:
I20250116 02:59:27.614910 21334 rpcz_store.cc:276] 0116 02:59:26.195858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.609669 (+1413811us) service_pool.cc:225] Handling call
0116 02:59:27.614536 (+ 4867us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.621248 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:27.625818 22947 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.628000 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:27.628839 20607 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:59:27.629266 20607 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 1983) took 1988 ms (client timeout 1984 ms). Trace:
W20250116 02:59:27.629614 20607 rpcz_store.cc:269] 0116 02:59:25.641087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.617637 (+1976550us) service_pool.cc:225] Handling call
0116 02:59:27.629192 (+ 11555us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.631620 20603 raft_consensus.cc:1235] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:27.631544 20720 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.538122 20754 rpcz_store.cc:276] 0116 02:59:26.143173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.761423 (+618250us) service_pool.cc:225] Handling call
0116 02:59:27.051419 (+289996us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.051735 (+ 316us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:27.052313 (+ 578us) log.cc:844] Serialized 14 byte log entry
0116 02:59:27.499604 (+447291us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.500143 (+ 539us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:27.504265 (+ 4122us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.515777 (+ 11512us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.520656 (+ 4879us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":207,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":10285,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":1435,"thread_start_us":428769,"threads_started":1,"wal-append.queue_time_us":440977}
I20250116 02:59:27.598409 20572 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:27.638876 23034 log.cc:926] Time spent T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.070s user 0.003s sys 0.000s
W20250116 02:59:27.655202 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 910475us
I20250116 02:59:27.650863 22989 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:27.658089 22989 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:27.660918 23039 consensus_queue.cc:1046] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:27.661947 23039 consensus_queue.cc:1225] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer 57fcfcd5ab01438c9a4baf8a61871aa6 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250116 02:59:27.664301 21330 raft_consensus.cc:1235] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:27.655965 22943 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:27.669669 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:27.670439 20601 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:27.640200 23011 log.cc:926] Time spent T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.569s user 0.005s sys 0.007s
W20250116 02:59:27.672055 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 85230.1us
I20250116 02:59:27.670334 20720 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:27.656968 22843 raft_consensus.cc:2743] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:27.677160 23039 consensus_queue.cc:1046] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:27.674485 22989 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:27.681501 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.672819 22947 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:27.644438 22612 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:27.683504 23014 log.cc:926] Time spent T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.614s user 0.003s sys 0.004s
I20250116 02:59:27.684077 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:27.684902 22997 log.cc:926] Time spent T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 1.510s user 0.015s sys 0.008s
I20250116 02:59:27.684556 22957 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.612561 20674 rpcz_store.cc:276] 0116 02:59:23.573711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:24.124278 (+550567us) service_pool.cc:225] Handling call
0116 02:59:24.124734 (+ 456us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:24.135481 (+ 10747us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:24.135516 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:24.136072 (+ 556us) log.cc:844] Serialized 26 byte log entry
0116 02:59:24.771008 (+634936us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:24.813940 (+ 42932us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:24.822079 (+ 8139us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:27.604743 (+2782664us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:27.606930 (+ 2187us) raft_consensus.cc:1680] finished
0116 02:59:27.606960 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.611989 (+ 5029us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10211,"raft.queue_time_us":11514,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":14217,"spinlock_wait_cycles":7106816,"thread_start_us":620098,"threads_started":1,"wal-append.queue_time_us":620659}
I20250116 02:59:27.685786 22715 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:27.700155 20674 raft_consensus.cc:1270] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:59:27.700868 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1542) took 3010 ms. Trace:
I20250116 02:59:27.684806 20603 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:27.707150 20720 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.708787 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:27.711444 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 84958.7us
I20250116 02:59:27.701246 20674 rpcz_store.cc:276] 0116 02:59:24.690704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.699331 (+3008627us) service_pool.cc:225] Handling call
0116 02:59:27.699747 (+ 416us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.700562 (+ 815us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.700785 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.714386 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:27.714780 22843 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:59:27.715348 20674 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:27.707289 20572 leader_election.cc:336] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.225s (SENT)
I20250116 02:59:27.717800 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:27.717986 20572 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:27.718485 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 403) took 1579 ms. Trace:
I20250116 02:59:27.719026 20756 rpcz_store.cc:276] 0116 02:59:26.139067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.139201 (+ 134us) service_pool.cc:225] Handling call
0116 02:59:26.187404 (+ 48203us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:26.214443 (+ 27039us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:26.215134 (+ 691us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:26.215859 (+ 725us) log.cc:844] Serialized 52 byte log entry
0116 02:59:26.509015 (+293156us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:26.509065 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:26.509145 (+ 80us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:27.592181 (+1083036us) spinlock_profiling.cc:243] Waited 92.6 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:27.712528 (+120347us) raft_consensus.cc:1680] finished
0116 02:59:27.712550 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.718411 (+ 5861us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":39400,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":24517,"spinlock_wait_cycles":212873984,"thread_start_us":172478,"threads_started":1,"wal-append.queue_time_us":814300}
I20250116 02:59:27.712066 20599 raft_consensus.cc:1235] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 5. Ops: []
I20250116 02:59:27.700974 22715 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:27.712366 22612 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:27.726658 20720 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.686439 22846 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:27.737618 22846 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:27.729486 22824 consensus_queue.cc:1046] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20250116 02:59:27.721271 22920 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:27.729624 22612 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:27.737138 22871 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:27.738440 20647 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.110s (SENT)
I20250116 02:59:27.733372 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 758) took 1429 ms. Trace:
I20250116 02:59:27.741307 20603 raft_consensus.cc:2388] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:59:27.735220 20753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1473) took 1984 ms (client timeout 1952 ms). Trace:
I20250116 02:59:27.742146 21336 rpcz_store.cc:276] 0116 02:59:26.303760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.335377 (+ 31617us) service_pool.cc:225] Handling call
0116 02:59:26.349562 (+ 14185us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:26.397026 (+ 47464us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:26.397065 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:26.397121 (+ 56us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:26.397167 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.733081 (+1335914us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b5400612bd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:27.733145 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.733274 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108551424}
I20250116 02:59:27.724737 22947 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250116 02:59:27.742411 20753 rpcz_store.cc:269] 0116 02:59:25.750488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:25.767269 (+ 16781us) service_pool.cc:225] Handling call
0116 02:59:27.734976 (+1967707us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540033c7d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:27.735138 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2985929984}
W20250116 02:59:27.745440 20647 leader_election.cc:336] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.219s (SENT)
I20250116 02:59:27.745358 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:27.740507 22895 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 3.043s user 0.016s sys 0.055s
I20250116 02:59:27.724967 20675 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:27.748440 20572 leader_election.cc:336] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.534s (SENT)
I20250116 02:59:27.743909 22612 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:27.752797 20572 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:27.762821 22715 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:27.763056 22871 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:27.744663 22947 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:27.771159 22947 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:27.775466 21336 raft_consensus.cc:1235] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
W20250116 02:59:27.751621 22895 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 3.053s user 0.016s sys 0.055s
I20250116 02:59:27.746860 22846 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:27.786252 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:27.787921 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 346) took 3546 ms. Trace:
I20250116 02:59:27.794736 20678 rpcz_store.cc:276] 0116 02:59:24.241008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.605883 (+3364875us) service_pool.cc:225] Handling call
0116 02:59:27.607874 (+ 1991us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:27.608097 (+ 223us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.608113 (+ 16us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:27.620868 (+ 12755us) log.cc:844] Serialized 26 byte log entry
0116 02:59:27.726537 (+105669us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:27.726580 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.726685 (+ 105us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:27.787644 (+ 60959us) raft_consensus.cc:1680] finished
0116 02:59:27.787672 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.787842 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":56864,"threads_started":1,"wal-append.queue_time_us":111636}
I20250116 02:59:27.796701 20602 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:27.797544 22927 raft_consensus.cc:2743] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:27.787988 22943 consensus_queue.cc:1046] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20250116 02:59:27.798141 20679 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:27.798795 22927 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250116 02:59:27.796787 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:27.800114 20678 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:27.800786 22957 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.798851 22633 raft_consensus.cc:3049] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:27.790074 22914 raft_consensus.cc:3049] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:27.801580 22633 raft_consensus.cc:738] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:27.803035 22914 raft_consensus.cc:738] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:27.803746 22821 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:27.804117 22914 consensus_queue.cc:260] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:27.805450 22914 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:27.807271 20647 leader_election.cc:336] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.636s (SENT)
I20250116 02:59:27.807806 20647 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:27.809466 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 61037.6us
W20250116 02:59:27.811883 20647 leader_election.cc:336] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.888s (SENT)
I20250116 02:59:27.797610 21331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 447) took 1411 ms. Trace:
I20250116 02:59:27.815315 21331 rpcz_store.cc:276] 0116 02:59:26.386474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.414272 (+1027798us) service_pool.cc:225] Handling call
0116 02:59:27.506561 (+ 92289us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:27.558019 (+ 51458us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.797388 (+239369us) spinlock_profiling.cc:243] Waited 89.1 ms on lock 0x7b540048fb50. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:27.797547 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":204848896}
I20250116 02:59:27.774139 22897 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.811688 22943 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:27.821463 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:27.825186 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 448) took 1438 ms. Trace:
I20250116 02:59:27.784652 22994 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.831298 20678 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:27.833482 21335 rpcz_store.cc:276] 0116 02:59:26.386842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.423302 (+1036460us) service_pool.cc:225] Handling call
0116 02:59:27.436543 (+ 13241us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.575839 (+139296us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.824923 (+249084us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54005c3ad4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:27.825114 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":318818560}
I20250116 02:59:27.836932 20676 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:27.872803 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:27.820297 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 93724.1us
I20250116 02:59:27.873960 20720 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.841343 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 345) took 3600 ms. Trace:
I20250116 02:59:27.878916 20571 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:27.876538 20682 rpcz_store.cc:276] 0116 02:59:24.240684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.068428 (+1827744us) service_pool.cc:225] Handling call
0116 02:59:26.120497 (+ 52069us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:26.254115 (+133618us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.841037 (+1586922us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400087ad4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:27.841267 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":305613056}
I20250116 02:59:27.835726 20602 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:27.828706 21331 raft_consensus.cc:1235] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
W20250116 02:59:27.882598 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 48557.2us
I20250116 02:59:27.839031 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:27.882321 23038 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:27.844785 22924 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:27.892654 21302 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.826351 20679 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:27.910586 21302 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:27.911321 20679 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1540) took 4332 ms (client timeout 2204 ms). Trace:
I20250116 02:59:27.911438 22927 consensus_queue.cc:1046] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:59:27.896919 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:27.911790 20679 rpcz_store.cc:269] 0116 02:59:23.578462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.799012 (+220550us) service_pool.cc:225] Handling call
0116 02:59:27.797664 (+3998652us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b5800680fb0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326c8b1c3 00007fc326ceeaba 00007fc326d063af 00007fc326d07b4e 00007fc326d13228 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c
0116 02:59:27.911218 (+113554us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4926688128}
I20250116 02:59:27.913473 20678 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:27.833210 22846 consensus_queue.cc:1046] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20250116 02:59:27.913754 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 348) took 3222 ms. Trace:
I20250116 02:59:27.914054 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 347) took 3222 ms. Trace:
I20250116 02:59:27.885637 22943 consensus_queue.cc:1046] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20250116 02:59:27.914700 22868 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:27.915148 22868 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:27.915480 22868 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:27.915730 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:27.838640 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:27.917043 21330 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:27.917641 22715 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:27.918567 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:27.920416 22895 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 02:59:27.920884 22980 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:27.921491 22980 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:27.922140 22881 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250116 02:59:27.923043 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 733) took 4480 ms. Trace:
I20250116 02:59:27.918185 22715 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.925513 22927 raft_consensus.cc:3049] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:27.925917 22927 raft_consensus.cc:738] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:27.836546 22897 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.834695 22633 consensus_queue.cc:260] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.838486 22960 raft_consensus.cc:3049] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:27.927886 22960 raft_consensus.cc:738] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:27.883189 23048 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:27.928465 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.928870 22960 consensus_queue.cc:260] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.929553 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.926999 22927 consensus_queue.cc:260] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:27.930382 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1553) took 1303 ms. Trace:
I20250116 02:59:27.899590 20682 raft_consensus.cc:1235] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250116 02:59:27.903540 20720 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:27.927538 22633 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:27.920708 22866 raft_consensus.cc:3049] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
W20250116 02:59:27.837121 23017 log.cc:926] Time spent T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.903s user 0.005s sys 0.002s
I20250116 02:59:27.885674 22903 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20250116 02:59:27.836236 22998 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:27.889437 20649 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:27.931756 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:27.930902 22927 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:27.952523 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 63180.2us
I20250116 02:59:27.876523 22825 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:27.939450 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1543) took 3248 ms. Trace:
I20250116 02:59:27.930752 20681 rpcz_store.cc:276] 0116 02:59:26.626653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.106000 (+479347us) service_pool.cc:225] Handling call
0116 02:59:27.929388 (+823388us) spinlock_profiling.cc:243] Waited 9.38 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:27.930317 (+ 929us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21566720}
I20250116 02:59:27.944075 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:27.926296 22715 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:27.948695 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 45238.3us
I20250116 02:59:27.888521 23025 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:27.925829 23038 raft_consensus.cc:695] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:27.926362 22600 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:27.914423 20678 rpcz_store.cc:276] 0116 02:59:24.691428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.912687 (+3221259us) service_pool.cc:225] Handling call
0116 02:59:27.913093 (+ 406us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.913823 (+ 730us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.914001 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32384}
W20250116 02:59:27.950632 20683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1530) took 4474 ms (client timeout 1903 ms). Trace:
I20250116 02:59:27.974918 20680 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250116 02:59:27.918172 22947 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:27.923450 21329 rpcz_store.cc:276] 0116 02:59:23.442472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.463210 (+ 20738us) service_pool.cc:225] Handling call
0116 02:59:24.451546 (+988336us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:24.469512 (+ 17966us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:24.705884 (+236372us) raft_consensus.cc:1529] Early marking committed up to index 5
0116 02:59:25.733370 (+1027486us) log.cc:844] Serialized 14 byte log entry
0116 02:59:26.122072 (+388702us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:26.128916 (+ 6844us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:26.137648 (+ 8732us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.922737 (+1785089us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b540076a7d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:27.922805 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.922951 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3470,"raft.queue_time_us":52588,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":13216,"spinlock_wait_cycles":508790784,"thread_start_us":319794,"threads_started":1,"wal-append.queue_time_us":400887,"wal-append.run_cpu_time_us":1363,"wal-append.run_wall_time_us":1079746}
I20250116 02:59:27.953050 20677 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:27.834571 22957 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.106s
I20250116 02:59:27.954777 22766 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:27.957346 22953 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:27.952123 22846 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:27.950556 23047 raft_consensus.cc:2743] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:27.943049 22866 raft_consensus.cc:738] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:27.967200 22821 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:27.967599 21330 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:27.953095 20607 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:27.969650 20682 rpcz_store.cc:276] 0116 02:59:24.691044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.883702 (+3192658us) service_pool.cc:225] Handling call
0116 02:59:27.899311 (+ 15609us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:27.939080 (+ 39769us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.939376 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.999235 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 354) took 2313 ms. Trace:
I20250116 02:59:28.002759 20678 rpcz_store.cc:276] 0116 02:59:25.685466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.978974 (+2293508us) service_pool.cc:225] Handling call
0116 02:59:27.998523 (+ 19549us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.998835 (+ 312us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.998861 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.998914 (+ 53us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:27.998962 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.999024 (+ 62us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.999158 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.972080 20649 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:27.974843 20683 rpcz_store.cc:269] 0116 02:59:23.471365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:23.471483 (+ 118us) service_pool.cc:225] Handling call
0116 02:59:27.945452 (+4473969us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54002cb854. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:27.945625 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2832681728}
I20250116 02:59:27.980449 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 350) took 3260 ms. Trace:
I20250116 02:59:27.983168 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 349) took 3291 ms. Trace:
I20250116 02:59:27.945775 22936 consensus_queue.cc:1046] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20250116 02:59:27.997416 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:28.005218 20681 rpcz_store.cc:276] 0116 02:59:24.692021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.970165 (+3278144us) service_pool.cc:225] Handling call
0116 02:59:27.978937 (+ 8772us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.982695 (+ 3758us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.982734 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.982788 (+ 54us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:27.982846 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.982923 (+ 77us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.983090 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":16105}
I20250116 02:59:27.979058 22518 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:28.012758 20681 raft_consensus.cc:1235] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:28.011411 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:28.010398 20680 rpcz_store.cc:276] 0116 02:59:24.715560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.973088 (+3257528us) service_pool.cc:225] Handling call
0116 02:59:27.973416 (+ 328us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:27.975515 (+ 2099us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.975727 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:28.017143 20680 raft_consensus.cc:1270] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:28.019202 20678 raft_consensus.cc:1235] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:28.019865 21302 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.021623 20646 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:28.027314 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1489) took 1648 ms. Trace:
I20250116 02:59:28.027769 20681 rpcz_store.cc:276] 0116 02:59:26.378468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.012176 (+1633708us) service_pool.cc:225] Handling call
0116 02:59:28.012555 (+ 379us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.027019 (+ 14464us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.027248 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:28.025532 22989 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:27.977437 22825 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.001260 22821 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:28.020452 20649 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.032021 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1493) took 1629 ms. Trace:
I20250116 02:59:28.045603 20680 rpcz_store.cc:276] 0116 02:59:26.402591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.016306 (+1613715us) service_pool.cc:225] Handling call
0116 02:59:28.016702 (+ 396us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.031706 (+ 15004us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.031952 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:27.980890 22895 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20250116 02:59:27.934937 20679 rpcz_store.cc:276] 0116 02:59:24.691719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.912708 (+3220989us) service_pool.cc:225] Handling call
0116 02:59:27.913123 (+ 415us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.913352 (+ 229us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.913382 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:27.913429 (+ 47us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:27.913500 (+ 71us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.913563 (+ 63us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:27.913683 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65664}
I20250116 02:59:28.030759 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 355) took 1780 ms. Trace:
I20250116 02:59:28.034870 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:27.979095 22600 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:28.022725 22871 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:28.000505 22953 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.048352 20678 rpcz_store.cc:276] 0116 02:59:26.250020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.003742 (+1753722us) service_pool.cc:225] Handling call
0116 02:59:28.019037 (+ 15295us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.030364 (+ 11327us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.030678 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:28.030079 22879 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:27.995703 22766 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:27.997350 22846 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.074970 20607 raft_consensus.cc:2405] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250116 02:59:28.063890 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 92584.5us
I20250116 02:59:28.048660 23038 consensus_queue.cc:237] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:28.050266 22821 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:28.042182 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:28.088676 20647 leader_election.cc:336] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.762s (SENT)
I20250116 02:59:28.069212 22943 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:28.048326 22989 raft_consensus.cc:695] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:28.088745 20720 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.090580 20647 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:28.023041 22518 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.073719 22871 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:28.095005 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:28.049615 20674 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:28.072763 22994 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:28.019202 22868 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.100365 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 356) took 1850 ms. Trace:
I20250116 02:59:27.977166 23025 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.102663 22885 consensus_queue.cc:1046] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20250116 02:59:28.103255 20720 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:28.103556 23046 log.cc:926] Time spent T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20250116 02:59:28.104347 20601 raft_consensus.cc:2388] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:28.108268 20681 raft_consensus.cc:1235] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:28.117750 20720 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.119431 20571 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:28.119560 20647 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.838s (SENT) [suppressed 51 similar messages]
I20250116 02:59:28.124732 20720 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:28.106977 22932 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:28.105480 22927 raft_consensus.cc:2743] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:28.123695 22957 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:28.126406 21302 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.133921 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1632) took 1811 ms. Trace:
I20250116 02:59:28.123667 23047 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:28.112464 22871 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:28.134825 22895 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:28.135695 22957 raft_consensus.cc:695] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:28.137921 23047 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:28.097038 22903 consensus_queue.cc:1046] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20250116 02:59:28.120913 23054 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:28.140585 22846 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:28.069547 22866 consensus_queue.cc:260] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.141568 22825 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:28.141449 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:28.104297 22942 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:28.136696 22957 consensus_queue.cc:237] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:28.134351 20606 rpcz_store.cc:276] 0116 02:59:26.322398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.331641 (+ 9243us) service_pool.cc:225] Handling call
0116 02:59:26.343584 (+ 11943us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:26.473706 (+130122us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.133611 (+1659905us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b540014d2d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:28.133828 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269345664}
I20250116 02:59:28.142119 22989 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.121292 22600 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:28.161900 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:28.163692 20606 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:28.131090 23048 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:28.134852 22715 raft_consensus.cc:3049] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:28.132576 22947 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:28.216490 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 72740.9us
I20250116 02:59:28.223188 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1485) took 1994 ms. Trace:
I20250116 02:59:28.140679 22766 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:28.153158 23047 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:28.238225 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:28.239266 20606 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:28.250819 22908 consensus_queue.cc:1046] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:28.252346 22908 raft_consensus.cc:3049] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:28.252715 22908 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:28.255015 22866 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:28.255688 23047 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:28.240773 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:28.259634 20600 raft_consensus.cc:2371] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2. Re-sending same reply.
I20250116 02:59:28.261071 21302 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:28.262411 22884 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250116 02:59:28.263031 22884 raft_consensus.cc:695] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:28.264079 22884 consensus_queue.cc:237] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:28.268265 20749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1465) took 2603 ms (client timeout 1670 ms). Trace:
W20250116 02:59:28.268765 20749 rpcz_store.cc:269] 0116 02:59:25.664551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:25.673333 (+ 8782us) service_pool.cc:225] Handling call
0116 02:59:28.267996 (+2594663us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b540012dc54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:28.268167 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95293824}
I20250116 02:59:28.271178 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 748) took 2146 ms. Trace:
I20250116 02:59:28.271678 21327 rpcz_store.cc:276] 0116 02:59:26.124862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.125883 (+ 1021us) service_pool.cc:225] Handling call
0116 02:59:27.360823 (+1234940us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.361609 (+ 786us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.270934 (+909325us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540058f050. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:28.271101 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2799674240}
I20250116 02:59:28.274034 22993 raft_consensus.cc:3049] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:28.214591 20674 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:28.220116 22941 raft_consensus.cc:3049] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:28.275175 22941 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:28.275542 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:28.276530 22980 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:28.276077 22941 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.277602 22941 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:28.278040 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:28.278798 20600 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:28.279386 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:28.279925 20649 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:28.280203 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:28.280436 20571 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:28.287827 22519 raft_consensus.cc:2798] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:28.322039 22519 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:28.322587 22519 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:28.324851 23025 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:28.325462 23018 log.cc:926] Time spent T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 1.391s user 0.009s sys 0.000s
W20250116 02:59:28.327220 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 202539us
I20250116 02:59:28.329159 20720 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.330241 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:28.332125 20723 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:28.223371 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:28.153880 22612 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.336369 20571 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:28.338755 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.346028 20749 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:28.165109 22994 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20250116 02:59:28.344437 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:28.277007 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:28.223191 22715 raft_consensus.cc:738] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:28.326550 23048 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:28.358466 20647 leader_election.cc:336] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.005s (SENT)
I20250116 02:59:28.370168 20647 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.365084 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:28.274540 22993 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:28.371311 20571 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:28.371964 20606 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:28.381300 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.382489 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:28.373639 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1555) took 1267 ms. Trace:
I20250116 02:59:28.287977 22766 raft_consensus.cc:2798] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:28.333403 20675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1558) took 1205 ms. Trace:
I20250116 02:59:28.369834 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.383832 20680 rpcz_store.cc:276] 0116 02:59:27.106417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.053855 (+947438us) service_pool.cc:225] Handling call
0116 02:59:28.054321 (+ 466us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.054594 (+ 273us) raft_consensus.cc:1529] Early marking committed up to index 4
0116 02:59:28.332931 (+278337us) log.cc:844] Serialized 14 byte log entry
0116 02:59:28.373227 (+ 40296us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:28.373290 (+ 63us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:28.373347 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.373417 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:28.373568 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24569,"prepare.queue_time_us":38493,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":146,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"thread_start_us":283837,"threads_started":2}
I20250116 02:59:28.388429 20571 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:28.392082 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 759) took 2088 ms. Trace:
I20250116 02:59:28.392427 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:28.333189 20600 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:28.397006 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.344942 22947 raft_consensus.cc:2743] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:28.398500 20571 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:28.361095 22846 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:28.402379 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:28.392491 21333 rpcz_store.cc:276] 0116 02:59:26.304077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.411915 (+107838us) service_pool.cc:225] Handling call
0116 02:59:26.436527 (+ 24612us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:26.467558 (+ 31031us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:26.467590 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:26.481635 (+ 14045us) log.cc:844] Serialized 26 byte log entry
0116 02:59:26.845762 (+364127us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:26.845814 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:26.845883 (+ 69us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:28.324849 (+1478966us) spinlock_profiling.cc:243] Waited 73.5 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:28.355957 (+ 31108us) raft_consensus.cc:1680] finished
0116 02:59:28.355981 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:28.392009 (+ 36028us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8856,"raft.queue_time_us":21346,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":10173,"spinlock_wait_cycles":169011968,"thread_start_us":244133,"threads_started":1,"wal-append.queue_time_us":425959}
I20250116 02:59:28.405530 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:28.406342 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.404794 23038 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:28.381475 22874 raft_consensus.cc:2743] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:28.387282 22766 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:28.415320 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.253721 22908 consensus_queue.cc:260] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.399521 20748 raft_consensus.cc:1235] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:28.289372 22871 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:28.399362 21330 raft_consensus.cc:2388] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:28.393620 22825 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:28.391706 22995 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:28.427366 20751 raft_consensus.cc:2405] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250116 02:59:28.407591 21335 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:28.433410 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.399768 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:28.432597 23061 consensus_queue.cc:1035] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20250116 02:59:28.441299 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:28.425838 22766 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:28.422472 22846 raft_consensus.cc:695] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:28.448132 23021 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 1.318s user 0.007s sys 0.001s
I20250116 02:59:28.449096 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:28.453033 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1630) took 2131 ms. Trace:
I20250116 02:59:28.415843 23054 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:28.401858 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.415184 22868 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:28.462672 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 765) took 1702 ms. Trace:
I20250116 02:59:28.470695 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:28.441990 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:28.419680 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 359) took 1525 ms. Trace:
I20250116 02:59:28.484380 21327 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:28.485589 23060 log.cc:926] Time spent T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250116 02:59:28.472270 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.472693 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.403541 23063 consensus_queue.cc:1046] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.098s
I20250116 02:59:28.431430 22843 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:28.629148 20755 raft_consensus.cc:2363] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 2. Current term is 3.
I20250116 02:59:28.631731 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:28.674131 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 203516us
I20250116 02:59:28.677346 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.683876 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:28.681689 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:28.688513 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1492) took 2496 ms (client timeout 1765 ms). Trace:
I20250116 02:59:28.471730 20749 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:28.386788 23048 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.490628 20608 rpcz_store.cc:276] 0116 02:59:26.321621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.321757 (+ 136us) service_pool.cc:225] Handling call
0116 02:59:26.340620 (+ 18863us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:26.495180 (+154560us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.452689 (+1957509us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54002d8054. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:28.452952 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":297202944}
W20250116 02:59:28.689040 20752 rpcz_store.cc:269] 0116 02:59:26.191836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.298715 (+106879us) service_pool.cc:225] Handling call
0116 02:59:28.688257 (+2389542us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b5400370fd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:28.688415 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":392213376}
I20250116 02:59:28.494952 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 362) took 1139 ms. Trace:
I20250116 02:59:28.496366 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:28.690330 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.388582 20675 rpcz_store.cc:276] 0116 02:59:27.122029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.359961 (+237932us) service_pool.cc:225] Handling call
0116 02:59:27.681290 (+321329us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:28.283373 (+602083us) spinlock_profiling.cc:243] Waited 38.6 ms on lock 0x7b5400285d54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:28.326818 (+ 43445us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91823232}
I20250116 02:59:28.561350 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.696624 20756 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
W20250116 02:59:28.475013 22914 consensus_meta.cc:287] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.664s user 0.012s sys 0.002s
W20250116 02:59:28.567979 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 207427us
I20250116 02:59:28.703811 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.423964 23038 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:28.494289 23058 log.cc:926] Time spent T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.269s user 0.001s sys 0.000s
I20250116 02:59:28.578406 21334 rpcz_store.cc:276] 0116 02:59:26.760419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.615627 (+855208us) service_pool.cc:225] Handling call
0116 02:59:27.616737 (+ 1110us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:27.623579 (+ 6842us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.623612 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:27.624175 (+ 563us) log.cc:844] Serialized 26 byte log entry
0116 02:59:28.454480 (+830305us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b540060ff20. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c94d50 00007fc326c9f288 00007fc326d0fa7b 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc
0116 02:59:28.454552 (+ 72us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:28.454635 (+ 83us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.454738 (+ 103us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:28.454787 (+ 49us) raft_consensus.cc:1680] finished
0116 02:59:28.454800 (+ 13us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:28.462586 (+ 7786us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1676,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":142,"spinlock_wait_cycles":50309376,"thread_start_us":94029,"threads_started":1,"wal-append.queue_time_us":88873}
I20250116 02:59:28.711447 20723 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:28.715461 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.717377 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 347) took 2342 ms. Trace:
I20250116 02:59:28.719189 20756 raft_consensus.cc:2388] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:28.607470 22959 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.372s
I20250116 02:59:28.691004 21302 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.690778 20676 rpcz_store.cc:276] 0116 02:59:27.355275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.280302 (+925027us) service_pool.cc:225] Handling call
0116 02:59:28.280603 (+ 301us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.280811 (+ 208us) raft_consensus.cc:1529] Early marking committed up to index 7
0116 02:59:28.281149 (+ 338us) log.cc:844] Serialized 14 byte log entry
0116 02:59:28.490300 (+209151us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:28.490351 (+ 51us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:28.490402 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.491398 (+ 996us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:28.494823 (+ 3425us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19776,"prepare.queue_time_us":3881,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":59,"raft.queue_time_us":2284,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":68445,"thread_start_us":206552,"threads_started":1}
I20250116 02:59:28.693379 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.579198 22995 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:28.695645 20606 raft_consensus.cc:2405] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:28.455341 22993 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.437170 22715 consensus_queue.cc:260] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:28.630093 20600 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:28.738772 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.738477 22995 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.753885 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:28.754132 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:28.754423 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:28.754981 23068 log.cc:926] Time spent T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.286s user 0.000s sys 0.001s
I20250116 02:59:28.755460 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:28.758138 23064 log.cc:926] Time spent T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
I20250116 02:59:28.758975 23073 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:28.759594 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:28.760506 20606 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:28.760929 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:28.762502 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:28.762766 21299 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.763195 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:28.763978 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:28.764706 22868 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:28.766460 23048 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:28.767158 23048 consensus_queue.cc:1035] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:59:28.694041 21300 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:28.628453 22881 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:28.704654 20752 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:28.780035 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.682543 21328 raft_consensus.cc:2388] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:28.707135 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.710024 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.476423 22908 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:28.717845 20604 rpcz_store.cc:276] 0116 02:59:26.375363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.441861 (+ 66498us) service_pool.cc:225] Handling call
0116 02:59:26.442236 (+ 375us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:26.442435 (+ 199us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:26.442455 (+ 20us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:26.491751 (+ 49296us) log.cc:844] Serialized 26 byte log entry
0116 02:59:27.097739 (+605988us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:27.097781 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:27.097862 (+ 81us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:28.447318 (+1349456us) spinlock_profiling.cc:243] Waited 48.8 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:28.713395 (+266077us) raft_consensus.cc:1680] finished
0116 02:59:28.713423 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:28.717301 (+ 3878us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112290688,"thread_start_us":324194,"threads_started":1,"wal-append.queue_time_us":607621}
I20250116 02:59:28.678592 23072 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:28.700574 22612 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:28.735647 20749 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:28.759907 23073 raft_consensus.cc:695] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
W20250116 02:59:28.782075 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 88164.6us
I20250116 02:59:28.591377 22880 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:28.727638 22825 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250116 02:59:28.788523 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.750427 22995 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:28.793203 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:28.746351 22932 consensus_meta.cc:287] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.602s user 0.010s sys 0.000s
I20250116 02:59:28.736737 22879 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:28.741212 23061 consensus_queue.cc:1046] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20250116 02:59:28.812321 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.793363 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.789466 22895 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:28.801905 20754 raft_consensus.cc:2405] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:28.792305 22715 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:28.821364 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.839172 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.840075 20756 raft_consensus.cc:2371] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5. Re-sending same reply.
W20250116 02:59:28.840291 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 120503us
I20250116 02:59:28.792137 23072 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:28.843712 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.850479 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.851059 20752 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:28.851749 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:28.807209 21328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 754) took 2578 ms (client timeout 2033 ms). Trace:
I20250116 02:59:28.727458 22846 consensus_queue.cc:237] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.874171 20751 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:28.875924 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.885612 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.886435 20752 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:28.887365 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.888547 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.889997 20752 raft_consensus.cc:2405] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250116 02:59:28.892289 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.894943 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.896016 20752 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:28.897008 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:28.831236 22519 consensus_meta.cc:287] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.508s user 0.006s sys 0.006s
I20250116 02:59:28.877066 20723 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:28.909862 23061 raft_consensus.cc:3049] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:28.861090 20606 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:28.847600 22895 raft_consensus.cc:695] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:28.893118 21328 rpcz_store.cc:269] 0116 02:59:26.229230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:26.286685 (+ 57455us) service_pool.cc:225] Handling call
0116 02:59:28.448849 (+2162164us) spinlock_profiling.cc:243] Waited 7.85 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:28.801665 (+352816us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18060160}
I20250116 02:59:28.921470 23073 consensus_queue.cc:237] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:28.813578 22871 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:28.898067 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.898730 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.819761 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:28.788758 22881 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:28.847209 23067 log.cc:926] Time spent T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
I20250116 02:59:28.820183 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:28.838416 22959 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:28.911748 20723 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:28.891146 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.913252 20752 raft_consensus.cc:2405] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20250116 02:59:28.919194 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 235442us
I20250116 02:59:28.854367 23072 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:28.903566 22866 consensus_meta.cc:287] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.648s user 0.009s sys 0.003s
I20250116 02:59:28.924176 20649 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:29.056461 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 132399us
I20250116 02:59:28.919257 23061 raft_consensus.cc:738] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:28.922547 22612 consensus_queue.cc:1035] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
W20250116 02:59:28.936046 23069 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.254s user 0.001s sys 0.000s
I20250116 02:59:28.936663 21335 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:28.936954 22895 consensus_queue.cc:237] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.937778 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1563) took 1221 ms. Trace:
W20250116 02:59:28.940936 23047 consensus_meta.cc:287] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.685s user 0.010s sys 0.000s
I20250116 02:59:28.942937 20750 pending_rounds.cc:77] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 5
I20250116 02:59:29.080591 20750 pending_rounds.cc:99] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250116 02:59:28.943910 22960 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:28.945189 20753 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250116 02:59:28.936702 22920 raft_consensus.cc:3049] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:29.023376 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1566) took 1238 ms. Trace:
I20250116 02:59:29.084264 20676 rpcz_store.cc:276] 0116 02:59:27.785288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.724099 (+938811us) service_pool.cc:225] Handling call
0116 02:59:28.739995 (+ 15896us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.740228 (+ 233us) raft_consensus.cc:1529] Early marking committed up to index 2
0116 02:59:28.787999 (+ 47771us) log.cc:844] Serialized 14 byte log entry
0116 02:59:29.022996 (+234997us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:29.023055 (+ 59us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:29.023107 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.023177 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:29.023314 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4549,"prepare.queue_time_us":76511,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":5824,"raft.queue_time_us":2372,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":11743,"thread_start_us":234539,"threads_started":1}
I20250116 02:59:29.033380 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:29.086973 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:28.949005 23070 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:28.993484 22519 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:28.607137 20679 rpcz_store.cc:276] 0116 02:59:26.894554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.047455 (+1152901us) service_pool.cc:225] Handling call
0116 02:59:28.047811 (+ 356us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.068445 (+ 20634us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:28.117218 (+ 48773us) log.cc:844] Serialized 14 byte log entry
0116 02:59:28.369887 (+252669us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:28.386984 (+ 17097us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:28.391534 (+ 4550us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.399549 (+ 8015us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:28.415656 (+ 16107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28041,"prepare.queue_time_us":45117,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":106,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":14602,"thread_start_us":219819,"threads_started":1,"wal-append.queue_time_us":315052}
I20250116 02:59:29.069866 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:29.100661 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:29.101248 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:28.960837 23038 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.056670 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:29.081458 20750 raft_consensus.cc:2883] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:29.078578 20680 rpcz_store.cc:276] 0116 02:59:27.716764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.397892 (+681128us) service_pool.cc:225] Handling call
0116 02:59:28.398912 (+ 1020us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:28.419007 (+ 20095us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:28.419047 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:28.432075 (+ 13028us) log.cc:844] Serialized 26 byte log entry
0116 02:59:28.806902 (+374827us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:28.806949 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.807025 (+ 76us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:28.937512 (+130487us) raft_consensus.cc:1680] finished
0116 02:59:28.937537 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:28.937705 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7608,"raft.queue_time_us":21963,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":779,"thread_start_us":308471,"threads_started":1,"wal-append.queue_time_us":462468}
I20250116 02:59:29.087982 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:29.060693 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:29.056720 22791 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:29.110718 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.142561 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:29.074002 23080 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:29.080744 20674 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
W20250116 02:59:29.154547 20647 leader_election.cc:336] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.630s (SENT)
I20250116 02:59:29.162137 20647 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:29.146703 20676 raft_consensus.cc:2388] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:29.093540 22920 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:29.174285 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:29.175098 20756 raft_consensus.cc:2371] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4. Re-sending same reply.
I20250116 02:59:29.179612 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:29.165160 22791 raft_consensus.cc:695] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:29.068136 22959 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.184981 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:29.096201 22866 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
I20250116 02:59:29.143029 23070 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.142372 23038 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.093526 23047 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.173274 22980 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:29.187759 22846 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.217108 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:29.160931 23079 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.154937 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:29.208550 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 113869us
W20250116 02:59:29.212993 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 301292us
I20250116 02:59:29.197911 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:29.193823 22612 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:29.187930 22957 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.235051 20755 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:29.191350 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.230567 20756 raft_consensus.cc:2371] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4. Re-sending same reply.
W20250116 02:59:29.217008 23081 log.cc:926] Time spent T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20250116 02:59:29.236604 20723 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:29.213050 22949 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:29.242726 20723 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:29.249531 20720 connection.cc:232] RPC call timeout handler was delayed by -1.0323269990001336s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1693s for a response
I20250116 02:59:29.252030 20720 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:29.240748 23076 log.cc:926] Time spent T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.311s user 0.001s sys 0.000s
I20250116 02:59:29.233794 22749 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.237524 22612 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:29.271150 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 115335us
I20250116 02:59:29.203305 22791 consensus_queue.cc:237] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.167909 23080 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.221261 23038 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.232861 22868 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20250116 02:59:29.267233 22821 consensus_meta.cc:287] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.168s user 0.014s sys 0.000s
I20250116 02:59:29.247225 23071 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.166787 23061 consensus_queue.cc:260] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.228626 22846 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.275719 22612 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:29.318420 20749 raft_consensus.cc:2405] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:29.313993 22881 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.237542 22920 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.325241 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 366) took 1343 ms. Trace:
W20250116 02:59:29.340134 20647 leader_election.cc:336] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.747s (SENT)
I20250116 02:59:29.340713 20647 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:29.339226 20601 rpcz_store.cc:276] 0116 02:59:27.982232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.682538 (+700306us) service_pool.cc:225] Handling call
0116 02:59:28.707278 (+ 24740us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:28.709834 (+ 2556us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:28.709866 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:28.726488 (+ 16622us) log.cc:844] Serialized 26 byte log entry
0116 02:59:28.928050 (+201562us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:28.928147 (+ 97us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.928240 (+ 93us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:29.325002 (+396762us) raft_consensus.cc:1680] finished
0116 02:59:29.325025 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:29.325170 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26823,"raft.queue_time_us":7260,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":17335,"thread_start_us":201097,"threads_started":1,"wal-append.queue_time_us":348092}
W20250116 02:59:29.362453 23087 log.cc:926] Time spent T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250116 02:59:29.324204 22519 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.319470 23071 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.244491 22957 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:28.105907 20683 rpcz_store.cc:276] 0116 02:59:26.250293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.005083 (+1754790us) service_pool.cc:225] Handling call
0116 02:59:28.018717 (+ 13634us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.018938 (+ 221us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:28.034224 (+ 15286us) log.cc:844] Serialized 14 byte log entry
0116 02:59:28.100035 (+ 65811us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:28.100078 (+ 43us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:28.100116 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.100194 (+ 78us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:28.100308 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25206,"prepare.queue_time_us":17549,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":3827,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":6091,"thread_start_us":61600,"threads_started":1}
I20250116 02:59:29.371778 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:29.254352 22949 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.307104 22843 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:29.387413 20647 leader_election.cc:336] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.656s (SENT)
I20250116 02:59:29.388046 20647 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:29.392025 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.392782 20683 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:29.276520 22749 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.397137 21299 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:29.225975 23070 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:29.369025 22846 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.324656 23098 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:29.246580 23079 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.378293 22980 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:29.381938 22866 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:29.406230 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.314564 22821 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:29.412449 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 161229us
W20250116 02:59:29.422046 23092 log.cc:926] Time spent T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20250116 02:59:29.382035 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.395439 23061 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:29.381461 23102 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:29.432539 20720 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:29.445911 20678 raft_consensus.cc:2388] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:29.446244 20720 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:29.438601 22519 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:29.440343 22879 consensus_meta.cc:287] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.621s user 0.008s sys 0.004s
W20250116 02:59:29.455102 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 56689us
I20250116 02:59:29.434649 22959 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.440063 22871 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250116 02:59:29.456820 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 69455us
I20250116 02:59:29.439308 23047 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:29.451464 22908 consensus_meta.cc:287] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.650s user 0.012s sys 0.000s
I20250116 02:59:29.447922 22846 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:29.400453 22843 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:29.474784 23077 log.cc:926] Time spent T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.238s user 0.000s sys 0.001s
W20250116 02:59:29.475214 23084 log.cc:926] Time spent T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
W20250116 02:59:29.476531 23082 log.cc:926] Time spent T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.156s user 0.001s sys 0.001s
I20250116 02:59:29.445290 20675 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:29.432163 22980 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.460053 22879 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:29.475831 23107 raft_consensus.cc:2798] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:29.476231 23094 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:29.474295 23089 raft_consensus.cc:2798] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:29.504104 20751 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:29.513118 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1903) took 1020 ms. Trace:
I20250116 02:59:29.513532 20753 raft_consensus.cc:1270] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:29.513491 20751 rpcz_store.cc:276] 0116 02:59:28.492705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.101007 (+608302us) service_pool.cc:225] Handling call
0116 02:59:29.513048 (+412041us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:29.468178 22995 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:29.516352 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:29.475467 23105 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.515340 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:29.518560 20751 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:29.524549 20683 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:29.455727 22519 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.526204 20571 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:29.503491 22949 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.431149 23098 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:29.473841 22880 consensus_meta.cc:287] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.660s user 0.006s sys 0.006s
W20250116 02:59:29.533288 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 230866us
I20250116 02:59:29.495328 23047 consensus_queue.cc:1035] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
W20250116 02:59:29.533635 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 264026us
I20250116 02:59:29.539439 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 372) took 1547 ms. Trace:
I20250116 02:59:29.562781 20606 rpcz_store.cc:276] 0116 02:59:27.991614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.946839 (+955225us) service_pool.cc:225] Handling call
0116 02:59:28.947184 (+ 345us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.947401 (+ 217us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:29.217954 (+270553us) log.cc:844] Serialized 14 byte log entry
0116 02:59:29.538994 (+321040us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:29.539063 (+ 69us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:29.539129 (+ 66us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.539211 (+ 82us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:29.539364 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":524786,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":156447,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":5756,"thread_start_us":488794,"threads_started":3,"wal-append.queue_time_us":235220}
I20250116 02:59:29.514117 23094 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:29.514219 23107 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:29.577945 23025 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:29.579003 23086 log.cc:926] Time spent T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.259s user 0.001s sys 0.000s
I20250116 02:59:29.517963 23089 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:29.517350 23038 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.448452 23102 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:29.586014 20600 consensus_meta.cc:287] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.805s user 0.007s sys 0.003s
I20250116 02:59:29.627492 23107 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:29.551656 23080 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.586376 23071 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.583153 22749 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:29.535341 23105 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.532078 22995 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.628800 23094 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:29.693821 20647 leader_election.cc:336] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.095s (SENT)
W20250116 02:59:29.674000 22766 consensus_meta.cc:287] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 1.172s user 0.013s sys 0.002s
I20250116 02:59:29.554563 22846 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:29.583372 22871 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:29.638180 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 373) took 1646 ms. Trace:
I20250116 02:59:29.655990 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:29.694353 20647 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:29.695384 20606 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
W20250116 02:59:29.623811 22941 consensus_meta.cc:287] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.317s user 0.017s sys 0.000s
I20250116 02:59:29.701095 23106 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250116 02:59:29.681963 23072 consensus_meta.cc:287] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.602s user 0.009s sys 0.002s
W20250116 02:59:29.584929 20572 leader_election.cc:336] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.511s (SENT)
I20250116 02:59:29.586862 23096 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:29.581243 22821 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.675032 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:29.637462 23073 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:29.666882 23090 log.cc:926] Time spent T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.224s user 0.001s sys 0.001s
I20250116 02:59:29.717263 23105 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.717975 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 368) took 1734 ms. Trace:
I20250116 02:59:29.718380 20608 rpcz_store.cc:276] 0116 02:59:27.983528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.714492 (+730964us) service_pool.cc:225] Handling call
0116 02:59:28.714902 (+ 410us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:28.912730 (+197828us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:28.912761 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:28.920133 (+ 7372us) log.cc:844] Serialized 26 byte log entry
0116 02:59:29.107865 (+187732us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:29.107951 (+ 86us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.108101 (+ 150us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:29.717689 (+609588us) raft_consensus.cc:1680] finished
0116 02:59:29.717715 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:29.717894 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24241,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"thread_start_us":370546,"threads_started":2,"wal-append.queue_time_us":302863}
I20250116 02:59:29.710685 20572 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:29.814110 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 229214us
I20250116 02:59:29.817080 20723 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:29.582521 22980 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.580726 22843 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.583384 23079 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:29.696370 20647 leader_election.cc:336] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.101s (SENT)
W20250116 02:59:29.832172 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 135822us
I20250116 02:59:29.701963 23098 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.705178 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:29.705534 22953 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.700930 22871 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.666214 23025 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.707646 23115 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:29.701972 23106 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.701990 21330 raft_consensus.cc:2405] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250116 02:59:29.693051 23111 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:29.669296 23089 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:29.837968 23120 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:29.659940 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 213780us
I20250116 02:59:29.826601 23061 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:29.853492 20751 raft_consensus.cc:1212] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[3.5-3.5] Dedup: 3.5->[]
I20250116 02:59:29.719969 22995 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:29.720341 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:29.855458 20751 raft_consensus.cc:1235] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250116 02:59:29.821362 23126 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:29.622417 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.681017 23038 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:29.672780 23070 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:29.724385 21334 raft_consensus.cc:2388] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:29.710170 22766 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.708899 22957 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.681191 20600 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:29.730845 20603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 2000) took 2013 ms (client timeout 2122 ms). Trace:
I20250116 02:59:29.693279 23080 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:29.739710 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1922) took 1022 ms. Trace:
W20250116 02:59:29.743515 23103 log.cc:926] Time spent T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
I20250116 02:59:29.703078 22846 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.704643 23109 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:29.746320 23102 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.746598 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:29.746989 22872 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:29.749233 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1987) took 2129 ms. Trace:
I20250116 02:59:29.750487 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 409) took 3606 ms. Trace:
I20250116 02:59:29.757012 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 424) took 1417 ms. Trace:
I20250116 02:59:29.716018 23072 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.751653 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1569) took 1607 ms. Trace:
W20250116 02:59:29.766242 22715 consensus_meta.cc:287] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.927s user 0.013s sys 0.000s
I20250116 02:59:29.768359 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:29.888634 22612 consensus_meta.cc:287] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.513s user 0.012s sys 0.000s
I20250116 02:59:29.769837 21335 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250116 02:59:29.889108 22612 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.741250 22519 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:29.775640 20607 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 370) took 1791 ms (client timeout 1553 ms). Trace:
W20250116 02:59:29.890262 20607 rpcz_store.cc:269] 0116 02:59:27.991107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.023491 (+ 32384us) service_pool.cc:225] Handling call
0116 02:59:29.775376 (+1751885us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b540014e1d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:29.775551 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54993664}
I20250116 02:59:29.891211 22612 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.891516 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:29.891862 20756 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
W20250116 02:59:29.886734 20677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 360) took 2978 ms (client timeout 2597 ms). Trace:
I20250116 02:59:29.892237 20607 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:29.892364 20677 rpcz_store.cc:269] 0116 02:59:26.908823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.479103 (+570280us) service_pool.cc:225] Handling call
0116 02:59:29.886520 (+2407417us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b540016dfd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:29.886655 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1060416896}
I20250116 02:59:29.893357 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:29.774375 22960 consensus_meta.cc:287] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.680s user 0.001s sys 0.010s
I20250116 02:59:29.893628 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.778374 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1889) took 1348 ms. Trace:
I20250116 02:59:29.806131 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 371) took 1814 ms. Trace:
I20250116 02:59:29.694978 20601 rpcz_store.cc:276] 0116 02:59:27.991768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.345967 (+1354199us) service_pool.cc:225] Handling call
0116 02:59:29.387191 (+ 41224us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:29.387402 (+ 211us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:29.465994 (+ 78592us) log.cc:844] Serialized 14 byte log entry
0116 02:59:29.637761 (+171767us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:29.637833 (+ 72us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:29.637893 (+ 60us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.637970 (+ 77us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:29.638111 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":137043,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":6352,"raft.queue_time_us":104402,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":7995,"thread_start_us":154955,"threads_started":2,"wal-append.queue_time_us":225356}
I20250116 02:59:29.895031 20604 rpcz_store.cc:276] 0116 02:59:27.991434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.789613 (+798179us) service_pool.cc:225] Handling call
0116 02:59:28.790020 (+ 407us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:28.790294 (+ 274us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:28.790321 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:28.817127 (+ 26806us) log.cc:844] Serialized 26 byte log entry
0116 02:59:29.115887 (+298760us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:29.115922 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.115993 (+ 71us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:29.805829 (+689836us) raft_consensus.cc:1680] finished
0116 02:59:29.805856 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:29.806038 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":262696,"threads_started":1,"wal-append.queue_time_us":374744}
W20250116 02:59:29.837298 20647 leader_election.cc:336] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.014s (SENT)
I20250116 02:59:29.830247 22924 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:29.896782 20647 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:29.896806 20649 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:29.894876 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.894472 20753 rpcz_store.cc:276] 0116 02:59:28.429702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.192661 (+762959us) service_pool.cc:225] Handling call
0116 02:59:29.513096 (+320435us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:29.514069 (+ 973us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.778178 (+264109us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b5400044050. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:29.778311 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":619500416}
I20250116 02:59:29.852349 20720 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:29.899374 20607 raft_consensus.cc:2388] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
W20250116 02:59:29.713356 23093 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.233s user 0.000s sys 0.001s
W20250116 02:59:29.714309 23104 log.cc:926] Time spent T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.237s user 0.001s sys 0.000s
I20250116 02:59:29.720927 23073 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.855934 20608 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:29.718400 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:29.854163 23061 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:29.880404 20755 rpcz_store.cc:276] 0116 02:59:28.717438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.788189 (+ 70751us) service_pool.cc:225] Handling call
0116 02:59:29.739426 (+951237us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b5400396554. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:29.739627 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":977792384}
W20250116 02:59:29.879705 20603 rpcz_store.cc:269] 0116 02:59:27.717232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.717354 (+ 122us) service_pool.cc:225] Handling call
0116 02:59:29.730466 (+2013112us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b540010a254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:29.730773 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51537152}
I20250116 02:59:29.850847 23120 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:29.822662 20572 leader_election.cc:336] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.841s (SENT)
I20250116 02:59:29.916357 20572 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:29.917284 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 94682us
W20250116 02:59:29.917901 20572 leader_election.cc:336] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.639s (SENT)
I20250116 02:59:29.827507 23105 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:29.882213 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 377) took 1869 ms. Trace:
W20250116 02:59:29.899596 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 47327.3us
I20250116 02:59:29.866797 23111 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:29.900486 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 63246.1us
I20250116 02:59:29.900717 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.922333 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:29.896564 22959 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:29.902875 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:29.929442 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.903359 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:29.902918 20649 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:29.887116 20750 rpcz_store.cc:276] 0116 02:59:28.339192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.350213 (+ 11021us) service_pool.cc:225] Handling call
0116 02:59:28.942470 (+592257us) spinlock_profiling.cc:243] Waited 3.27 ms on lock 0x7b58001f0598. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:28.942719 (+ 249us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:29.115871 (+173152us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:29.115934 (+ 63us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:29.116447 (+ 513us) log.cc:844] Serialized 26 byte log entry
0116 02:59:29.436322 (+319875us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:29.436365 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.436454 (+ 89us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:29.755160 (+318706us) raft_consensus.cc:1680] finished
0116 02:59:29.755181 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:29.756939 (+ 1758us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7512448,"thread_start_us":319490,"threads_started":1,"wal-append.queue_time_us":415836}
I20250116 02:59:29.933388 20608 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:29.941871 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:29.942013 20604 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 1.
I20250116 02:59:29.940457 20601 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:29.881511 22846 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:29.909171 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:29.904211 22612 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:29.907348 22519 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:29.910723 23061 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:29.880095 23038 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.904646 23126 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.007s
I20250116 02:59:29.912108 23073 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:29.853363 23106 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.918304 20572 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:29.844676 23115 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:29.883284 23080 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.916132 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 368) took 1862 ms. Trace:
I20250116 02:59:29.926458 20600 rpcz_store.cc:276] 0116 02:59:28.012969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.420852 (+407883us) service_pool.cc:225] Handling call
0116 02:59:29.881296 (+1460444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:29.929396 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:29.944801 20603 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:29.910738 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:29.946224 20647 leader_election.cc:336] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.157s (SENT)
I20250116 02:59:29.943789 22884 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:29.889741 23072 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:29.940701 20677 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:30.003543 20647 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:29.947446 20608 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:29.907163 22980 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250116 02:59:29.974534 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 56669.1us
W20250116 02:59:30.011540 20572 leader_election.cc:336] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.894s (SENT)
I20250116 02:59:30.012097 20572 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.013486 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:29.954785 20720 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:30.015545 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 60836.3us
I20250116 02:59:29.978050 20571 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:29.976228 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1900) took 1484 ms. Trace:
W20250116 02:59:29.946797 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 44012.8us
I20250116 02:59:29.983201 20750 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250116 02:59:29.980708 20676 rpcz_store.cc:276] 0116 02:59:28.053782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.163509 (+1109727us) service_pool.cc:225] Handling call
0116 02:59:29.183944 (+ 20435us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:29.187131 (+ 3187us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:29.187162 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:29.197454 (+ 10292us) log.cc:844] Serialized 26 byte log entry
0116 02:59:29.267986 (+ 70532us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:29.268073 (+ 87us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.268277 (+ 204us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:29.915913 (+647636us) raft_consensus.cc:1680] finished
0116 02:59:29.915936 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:29.916071 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2749,"raft.queue_time_us":3325,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":15399,"thread_start_us":32649,"threads_started":1,"wal-append.queue_time_us":219019}
I20250116 02:59:30.016783 20753 rpcz_store.cc:276] 0116 02:59:28.491925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.906248 (+1414323us) service_pool.cc:225] Handling call
0116 02:59:29.944151 (+ 37903us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:29.947988 (+ 3837us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:29.948020 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:29.975925 (+ 27905us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:29.975986 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.976051 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:29.976166 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:29.939919 23129 log.cc:926] Time spent T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20250116 02:59:29.922729 22936 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:30.029477 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1904) took 1536 ms. Trace:
I20250116 02:59:30.041491 20750 rpcz_store.cc:276] 0116 02:59:28.492975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.941322 (+1448347us) service_pool.cc:225] Handling call
0116 02:59:29.982834 (+ 41512us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.017218 (+ 34384us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.029408 (+ 12190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:29.988061 20603 raft_consensus.cc:1235] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:29.888445 20674 rpcz_store.cc:276] 0116 02:59:28.143803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.276725 (+132922us) service_pool.cc:225] Handling call
0116 02:59:29.751459 (+1474734us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b54001c57d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:29.751584 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1640981376}
I20250116 02:59:29.945560 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 362) took 1967 ms. Trace:
I20250116 02:59:30.053541 20602 rpcz_store.cc:276] 0116 02:59:27.979882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.983857 (+ 3975us) service_pool.cc:225] Handling call
0116 02:59:29.746441 (+1762584us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98d0e 0000000000353db9 00007fc32bbf8a70 00007fc32bcbe834 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:29.939492 (+193051us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17932672}
I20250116 02:59:30.008262 22980 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:29.947844 22872 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:30.055575 22872 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.028560 20677 raft_consensus.cc:2388] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:29.887022 20748 rpcz_store.cc:276] 0116 02:59:26.143869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.042225 (+898356us) service_pool.cc:225] Handling call
0116 02:59:27.839064 (+796839us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b58001f0598. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:27.839291 (+ 227us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.457804 (+618513us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.750227 (+1292423us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b5400184cd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:29.750425 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1279143040}
I20250116 02:59:30.053543 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1905) took 1560 ms. Trace:
I20250116 02:59:30.050022 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:30.057996 20753 rpcz_store.cc:276] 0116 02:59:28.493234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.017761 (+1524527us) service_pool.cc:225] Handling call
0116 02:59:30.052903 (+ 35142us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.053161 (+ 258us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.053188 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.053243 (+ 55us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:30.053304 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.053358 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.053478 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.058749 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2008) took 1981 ms. Trace:
I20250116 02:59:29.928536 22715 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:30.037897 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 59945.6us
I20250116 02:59:30.045028 20649 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:29.982297 22995 raft_consensus.cc:2743] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:30.059643 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 113454us
I20250116 02:59:30.059144 20608 rpcz_store.cc:276] 0116 02:59:28.077258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.014767 (+1937509us) service_pool.cc:225] Handling call
0116 02:59:30.058196 (+ 43429us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.058395 (+ 199us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.058419 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.058459 (+ 40us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:30.058499 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.058550 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.058670 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.060772 20571 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:30.029074 22927 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:30.065440 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 375) took 2073 ms. Trace:
I20250116 02:59:29.974012 23116 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:30.061748 20571 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:29.928764 23105 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.073204 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2009) took 1941 ms. Trace:
I20250116 02:59:30.074725 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2006) took 1998 ms. Trace:
I20250116 02:59:29.901530 23109 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:29.885288 20599 rpcz_store.cc:276] 0116 02:59:27.619879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.625795 (+ 5916us) service_pool.cc:225] Handling call
0116 02:59:27.626152 (+ 357us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:27.724379 (+ 98227us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.748933 (+2024554us) spinlock_profiling.cc:243] Waited 76 ms on lock 0x7b5400092254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:29.749164 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174866816}
I20250116 02:59:30.141644 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.067878 20649 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:30.052954 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:30.054823 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.056607 22843 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:30.069036 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:30.074573 20608 rpcz_store.cc:276] 0116 02:59:28.132215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.061077 (+1928862us) service_pool.cc:225] Handling call
0116 02:59:30.072671 (+ 11594us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.072858 (+ 187us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.072889 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.072931 (+ 42us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:30.072973 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.073022 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.073141 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.078497 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.078398 20571 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.078281 20603 rpcz_store.cc:276] 0116 02:59:28.076651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.983764 (+1907113us) service_pool.cc:225] Handling call
0116 02:59:29.984108 (+ 344us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.070743 (+ 86635us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.074660 (+ 3917us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.152477 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:30.152750 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:30.065672 20720 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50646 dropped due to backpressure: service queue is full with 50 items
I20250116 02:59:30.153461 20603 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:29.973520 22959 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.048328 23079 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:30.069731 20647 leader_election.cc:336] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.838s (SENT)
W20250116 02:59:30.145421 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 77631.7us
I20250116 02:59:30.166221 20647 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.064910 22980 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:30.167840 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2005) took 2091 ms. Trace:
I20250116 02:59:30.037981 22871 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:30.175127 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 105431us
W20250116 02:59:30.176164 20647 leader_election.cc:336] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.177s (SENT)
I20250116 02:59:30.153189 20677 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:30.154772 20674 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:30.155510 23120 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:29.952380 23094 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.071136 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:30.156116 23121 log.cc:926] Time spent T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.442s user 0.000s sys 0.004s
I20250116 02:59:29.973783 23107 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.157022 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:30.158164 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.159448 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:30.161255 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 92146.5us
I20250116 02:59:30.064558 22957 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250116 02:59:30.233443 20750 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:30.162576 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 84273.5us
I20250116 02:59:30.067087 22998 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:30.165338 20606 raft_consensus.cc:2405] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250116 02:59:30.240249 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.166579 20649 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:30.242163 20572 leader_election.cc:336] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.952s (SENT)
W20250116 02:59:30.242386 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 75922.9us
I20250116 02:59:30.160408 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 1995) took 2479 ms. Trace:
I20250116 02:59:30.242687 20572 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.007726 22884 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.071121 22953 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:30.089721 22519 raft_consensus.cc:2798] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:30.168675 20607 rpcz_store.cc:276] 0116 02:59:28.076302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.983331 (+1907029us) service_pool.cc:225] Handling call
0116 02:59:30.005880 (+ 22549us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.006112 (+ 232us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:30.044687 (+ 38575us) log.cc:844] Serialized 14 byte log entry
0116 02:59:30.167446 (+122759us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.167499 (+ 53us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:30.167559 (+ 60us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.167626 (+ 67us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.167780 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":38375,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"thread_start_us":122363,"threads_started":1}
W20250116 02:59:30.178658 20647 leader_election.cc:336] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.194s (SENT)
I20250116 02:59:30.051422 22936 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.150151 23114 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:30.144482 23071 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:30.090226 23080 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.188205 20723 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:30.189776 21302 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:30.254120 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 66015us
W20250116 02:59:30.254774 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 65079.1us
I20250116 02:59:30.170387 23126 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:30.148356 20753 raft_consensus.cc:2405] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:30.180348 22755 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:30.244321 23131 log.cc:926] Time spent T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.266s user 0.001s sys 0.000s
I20250116 02:59:30.239609 23113 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:30.243996 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:30.246258 23107 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.242935 20605 rpcz_store.cc:276] 0116 02:59:27.681102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.686128 (+ 5026us) service_pool.cc:225] Handling call
0116 02:59:29.472298 (+1786170us) spinlock_profiling.cc:243] Waited 5.4 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:29.472552 (+ 254us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:29.504684 (+ 32132us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:29.504723 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:29.625596 (+120873us) log.cc:844] Serialized 26 byte log entry
0116 02:59:29.838196 (+212600us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:29.838249 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.838338 (+ 89us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:30.143695 (+305357us) raft_consensus.cc:1680] finished
0116 02:59:30.143719 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.160327 (+ 16608us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23372,"raft.queue_time_us":42303,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":11217,"spinlock_wait_cycles":12413312,"thread_start_us":165933,"threads_started":1,"wal-append.queue_time_us":252082}
I20250116 02:59:30.244068 20649 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:30.252305 20647 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.255420 21302 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:30.253322 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.255251 23116 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:30.259012 20683 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250116 02:59:30.260547 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.262976 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2011) took 2019 ms. Trace:
I20250116 02:59:30.236590 23120 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:30.280588 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 372) took 1917 ms. Trace:
I20250116 02:59:30.282021 20683 rpcz_store.cc:276] 0116 02:59:28.363120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.238149 (+1875029us) service_pool.cc:225] Handling call
0116 02:59:30.258641 (+ 20492us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:30.280258 (+ 21617us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.280516 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.280378 20608 rpcz_store.cc:276] 0116 02:59:28.243728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.251716 (+2007988us) service_pool.cc:225] Handling call
0116 02:59:30.262381 (+ 10665us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.262588 (+ 207us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.262653 (+ 65us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.262705 (+ 52us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:30.262746 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.262800 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.262910 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.256567 23114 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:30.259632 22927 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:30.180136 22924 consensus_queue.cc:1046] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20250116 02:59:30.236773 22957 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:30.291877 22957 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:30.292282 22924 raft_consensus.cc:3049] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:30.292632 22924 raft_consensus.cc:738] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:30.293431 22924 consensus_queue.cc:260] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:30.295265 23136 log.cc:926] Time spent T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20250116 02:59:30.295575 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:30.270416 21335 raft_consensus.cc:2388] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:29.948860 23111 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:30.267699 20572 leader_election.cc:336] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.959s (SENT)
I20250116 02:59:30.302631 20572 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:30.303920 20572 leader_election.cc:336] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.996s (SENT)
I20250116 02:59:30.275844 20571 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:30.304850 20572 leader_election.cc:336] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.005s (SENT)
I20250116 02:59:30.154260 20720 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50646 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1523)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1526)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1505)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1503)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1495)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1924)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1928)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1939)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1940)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1524)
W20250116 02:59:30.305830 20572 leader_election.cc:336] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.107s (SENT)
I20250116 02:59:30.271620 22519 raft_consensus.cc:695] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:30.188205 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.274227 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2012) took 2017 ms. Trace:
I20250116 02:59:30.173892 22766 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.284839 20608 raft_consensus.cc:1235] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:30.257144 23071 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:30.257045 22715 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:30.264101 22994 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:30.263549 23105 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.289796 23038 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.290212 22872 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:30.291276 23137 log.cc:926] Time spent T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20250116 02:59:30.270426 23072 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:30.296696 23120 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:30.350164 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 762) took 3590 ms. Trace:
W20250116 02:59:30.287927 23135 log.cc:926] Time spent T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.258s user 0.001s sys 0.000s
I20250116 02:59:30.235646 23054 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:30.293243 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2015) took 2015 ms. Trace:
I20250116 02:59:30.278842 22821 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:30.279671 23126 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:30.317420 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.308732 20572 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:30.254343 22953 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.288082 22998 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:30.332857 20605 rpcz_store.cc:276] 0116 02:59:28.256437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.273357 (+2016920us) service_pool.cc:225] Handling call
0116 02:59:30.273736 (+ 379us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.273920 (+ 184us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.273942 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.273983 (+ 41us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:30.274021 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.274073 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.274171 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.336266 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:30.297889 22927 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:30.308131 22924 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:30.289312 23114 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:30.340015 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2013) took 2072 ms. Trace:
I20250116 02:59:30.310434 22600 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:30.309419 23079 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:30.275393 23113 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:30.280043 23116 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:30.350576 21332 rpcz_store.cc:276] 0116 02:59:26.759584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.536547 (+776963us) service_pool.cc:225] Handling call
0116 02:59:27.536946 (+ 399us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:27.537312 (+ 366us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:27.537336 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:27.540230 (+ 2894us) log.cc:844] Serialized 26 byte log entry
0116 02:59:29.509865 (+1969635us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b1800302bc0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32698be4b 00007fc326991913 00007fc326991cc1 00007fc326ce345f
0116 02:59:29.829414 (+319549us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:29.829460 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:29.829575 (+ 115us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:30.343466 (+513891us) raft_consensus.cc:1680] finished
0116 02:59:30.343487 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.350065 (+ 6578us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":88,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"spinlock_wait_cycles":28621184,"thread_start_us":319267,"threads_started":1,"wal-append.queue_time_us":2375486}
I20250116 02:59:30.333285 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2010) took 2178 ms. Trace:
I20250116 02:59:30.340942 23071 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:30.338732 20748 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:30.353696 21330 raft_consensus.cc:2405] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250116 02:59:30.346153 23106 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:30.375442 22927 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:30.355989 23122 log.cc:926] Time spent T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.477s user 0.003s sys 0.010s
I20250116 02:59:30.372370 20607 raft_consensus.cc:2388] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:30.372126 20608 rpcz_store.cc:276] 0116 02:59:28.268065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.284221 (+2016156us) service_pool.cc:225] Handling call
0116 02:59:30.284638 (+ 417us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.339711 (+ 55073us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.339950 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:30.373154 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 67345.9us
I20250116 02:59:30.313978 22957 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:30.370535 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2016) took 2042 ms. Trace:
W20250116 02:59:30.378190 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 60893.4us
I20250116 02:59:30.358963 23054 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:30.355388 22994 raft_consensus.cc:695] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:30.387198 20720 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50646 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1526)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1505)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1503)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1495)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1924)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1928)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1939)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1940)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 439)
I20250116 02:59:30.387285 23094 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.397174 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:59:30.368640 22821 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:30.343210 23107 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:30.438807 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 260183us
I20250116 02:59:30.345702 22715 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.383409 23126 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:30.405639 22993 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:30.404114 23089 consensus_meta.cc:287] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.553s user 0.007s sys 0.004s
I20250116 02:59:30.466971 20605 rpcz_store.cc:276] 0116 02:59:28.327832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.369626 (+2041794us) service_pool.cc:225] Handling call
0116 02:59:30.369997 (+ 371us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.370203 (+ 206us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.370228 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.370268 (+ 40us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:30.370315 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.370370 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.370475 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.432981 22872 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:30.470817 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 196780us
I20250116 02:59:30.380360 23038 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250116 02:59:30.439793 23141 log.cc:926] Time spent T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.062s user 0.000s sys 0.002s
I20250116 02:59:30.384510 23116 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:30.438788 23139 log.cc:926] Time spent T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20250116 02:59:30.403769 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 427) took 1916 ms. Trace:
I20250116 02:59:30.475258 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:30.392529 22953 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.378887 22600 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:30.393559 23111 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.477716 20603 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 2.
I20250116 02:59:30.437961 22884 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:30.495362 22864 consensus_queue.cc:1046] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20250116 02:59:30.480667 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 381) took 1582 ms. Trace:
I20250116 02:59:30.520800 20605 rpcz_store.cc:276] 0116 02:59:28.898739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.479850 (+1581111us) service_pool.cc:225] Handling call
0116 02:59:30.480178 (+ 328us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.480362 (+ 184us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.480388 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.480428 (+ 40us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:30.480465 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.480510 (+ 45us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.480613 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.522586 20605 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250116 02:59:30.465693 20572 leader_election.cc:336] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.114s (SENT)
I20250116 02:59:30.523373 20572 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:30.524416 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 58781.7us
I20250116 02:59:30.384366 23120 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:30.394917 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:30.533175 21331 raft_consensus.cc:2388] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:30.533309 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 425) took 2193 ms. Trace:
I20250116 02:59:30.533705 20754 rpcz_store.cc:276] 0116 02:59:28.339511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.343797 (+ 4286us) service_pool.cc:225] Handling call
0116 02:59:30.533028 (+2189231us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b540033c7d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:30.533232 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":505438592}
I20250116 02:59:30.473412 22821 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:30.534967 21331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 506) took 2194 ms (client timeout 2262 ms). Trace:
I20250116 02:59:30.535003 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:30.535521 21331 rpcz_store.cc:269] 0116 02:59:28.341037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.344069 (+ 3032us) service_pool.cc:225] Handling call
0116 02:59:30.394683 (+2050614us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:30.534897 (+140214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8003328}
I20250116 02:59:30.538275 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 374) took 2546 ms. Trace:
I20250116 02:59:30.490577 20756 rpcz_store.cc:276] 0116 02:59:28.487401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.242112 (+754711us) service_pool.cc:225] Handling call
0116 02:59:29.243836 (+ 1724us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:29.892302 (+648466us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.403483 (+511181us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b5400185954. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:30.403694 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1489824128}
I20250116 02:59:30.538669 20604 rpcz_store.cc:276] 0116 02:59:27.991940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.942784 (+1950844us) service_pool.cc:225] Handling call
0116 02:59:29.971906 (+ 29122us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:29.973353 (+ 1447us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:29.973381 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:29.973788 (+ 407us) log.cc:844] Serialized 26 byte log entry
0116 02:59:30.238668 (+264880us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:30.238740 (+ 72us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.238824 (+ 84us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:30.538029 (+299205us) raft_consensus.cc:1680] finished
0116 02:59:30.538052 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.538195 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2356,"raft.queue_time_us":1703,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":190509,"threads_started":1,"wal-append.queue_time_us":336464}
I20250116 02:59:30.539623 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.540874 20604 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:30.540966 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.541453 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2018) took 1618 ms. Trace:
I20250116 02:59:30.541788 20756 raft_consensus.cc:2388] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:30.541797 20604 rpcz_store.cc:276] 0116 02:59:28.923248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.540199 (+1616951us) service_pool.cc:225] Handling call
0116 02:59:30.540521 (+ 322us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.541227 (+ 706us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.541398 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.476956 22936 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.477339 22999 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:30.527523 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:30.543116 22999 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:30.529242 20572 leader_election.cc:336] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.947s (SENT)
I20250116 02:59:30.544616 20572 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.544823 22791 raft_consensus.cc:3049] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:30.545215 22791 raft_consensus.cc:738] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:30.545743 20572 leader_election.cc:336] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.999s (SENT)
I20250116 02:59:30.379570 22519 consensus_queue.cc:237] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:30.529870 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 382) took 1616 ms. Trace:
I20250116 02:59:30.546190 20572 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.546401 20605 rpcz_store.cc:276] 0116 02:59:28.913845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.521798 (+1607953us) service_pool.cc:225] Handling call
0116 02:59:30.522197 (+ 399us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.522981 (+ 784us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.529796 (+ 6815us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.548529 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 371) took 2223 ms. Trace:
W20250116 02:59:30.549212 20572 leader_election.cc:336] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.198s (SENT)
I20250116 02:59:30.549705 20572 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:30.550815 20572 leader_election.cc:336] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.204s (SENT)
I20250116 02:59:30.551235 20572 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:30.552073 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:30.552314 20572 leader_election.cc:336] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.304s (SENT)
I20250116 02:59:30.552788 20675 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:30.552842 20572 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.066123 20601 rpcz_store.cc:276] 0116 02:59:27.992090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.943609 (+1951519us) service_pool.cc:225] Handling call
0116 02:59:29.974834 (+ 31225us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:29.975131 (+ 297us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:29.975462 (+ 331us) log.cc:844] Serialized 14 byte log entry
0116 02:59:30.065040 (+ 89578us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.065097 (+ 57us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:30.065163 (+ 66us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.065266 (+ 103us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.065383 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":5331,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":1707,"thread_start_us":89174,"threads_started":1}
I20250116 02:59:30.557512 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.559170 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 369) took 2485 ms. Trace:
I20250116 02:59:30.559609 20678 rpcz_store.cc:276] 0116 02:59:28.073468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.082918 (+ 9450us) service_pool.cc:225] Handling call
0116 02:59:30.558971 (+2476053us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54001c66d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:30.559095 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3082072960}
I20250116 02:59:30.560089 22884 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:30.562013 22766 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:30.562656 22766 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:30.564054 20504 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41956 (request call id 335) took 18412 ms (client timeout 15000 ms). Trace:
I20250116 02:59:30.565438 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1899) took 2076 ms. Trace:
I20250116 02:59:30.565814 20751 rpcz_store.cc:276] 0116 02:59:28.489414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.531750 (+1042336us) service_pool.cc:225] Handling call
0116 02:59:29.532160 (+ 410us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:29.855950 (+323790us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.565187 (+709237us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b54000cdad4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:30.565375 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":738623872}
I20250116 02:59:30.567992 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.568821 20751 raft_consensus.cc:2388] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:30.569535 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:30.569903 23025 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.570113 20751 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:30.535660 23038 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:30.482348 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:59:30.553480 23105 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:30.575773 23142 log.cc:926] Time spent T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20250116 02:59:30.576602 23088 log.cc:926] Time spent T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 1.271s user 0.005s sys 0.006s
I20250116 02:59:30.577153 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 370) took 2433 ms. Trace:
I20250116 02:59:30.548749 20605 raft_consensus.cc:1235] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:30.545145 22999 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.551262 23079 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:30.577605 20676 rpcz_store.cc:276] 0116 02:59:28.143465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.058913 (+1915448us) service_pool.cc:225] Handling call
0116 02:59:30.143796 (+ 84883us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:30.144071 (+ 275us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.144098 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:30.146233 (+ 2135us) log.cc:844] Serialized 26 byte log entry
0116 02:59:30.332777 (+186544us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:30.332820 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.332888 (+ 68us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:30.576891 (+244003us) raft_consensus.cc:1680] finished
0116 02:59:30.576912 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.577088 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1314176,"thread_start_us":186053,"threads_started":1,"wal-append.queue_time_us":329771}
I20250116 02:59:30.581823 20676 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:30.582218 21300 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:30.582372 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1502) took 1825 ms. Trace:
I20250116 02:59:30.548916 20677 rpcz_store.cc:276] 0116 02:59:28.324714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.177824 (+1853110us) service_pool.cc:225] Handling call
0116 02:59:30.183248 (+ 5424us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.378680 (+195432us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.378724 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.378767 (+ 43us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:30.378811 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.548313 (+169502us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54002a4ed4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:30.548359 (+ 46us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.548469 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":448875264}
I20250116 02:59:30.583174 20676 rpcz_store.cc:276] 0116 02:59:28.757330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.581316 (+1823986us) service_pool.cc:225] Handling call
0116 02:59:30.581653 (+ 337us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.582151 (+ 498us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.582320 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.490849 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:30.591782 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:30.592003 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.592617 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1496) took 4148 ms. Trace:
I20250116 02:59:30.593264 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.593932 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1649) took 1033 ms. Trace:
I20250116 02:59:30.476931 23107 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.595808 20601 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:30.596457 20681 rpcz_store.cc:276] 0116 02:59:26.444160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.029006 (+1584846us) service_pool.cc:225] Handling call
0116 02:59:28.108047 (+ 79041us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.115349 (+ 7302us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.583200 (+2467851us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b540016d0d0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:30.592547 (+ 9347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":177516288}
I20250116 02:59:30.590747 22753 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:30.596901 20605 rpcz_store.cc:276] 0116 02:59:29.560058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.548343 (+988285us) service_pool.cc:225] Handling call
0116 02:59:30.548589 (+ 246us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.593668 (+ 45079us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.593873 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.486227 23089 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.359659 20602 rpcz_store.cc:276] 0116 02:59:28.278074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.292096 (+2014022us) service_pool.cc:225] Handling call
0116 02:59:30.292570 (+ 474us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.292813 (+ 243us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.292844 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.292913 (+ 69us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:30.292951 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.293012 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.293167 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.601042 21302 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:30.612361 20681 raft_consensus.cc:1270] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250116 02:59:30.409430 20599 rpcz_store.cc:276] 0116 02:59:28.155422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.164374 (+2008952us) service_pool.cc:225] Handling call
0116 02:59:30.257247 (+ 92873us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.272690 (+ 15443us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.275103 (+ 2413us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.281463 (+ 6360us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:30.287577 (+ 6114us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.300105 (+ 12528us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.319728 (+ 19623us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.491523 22872 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.615662 20754 raft_consensus.cc:2388] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:30.519268 22600 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:30.613014 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1574) took 1542 ms. Trace:
I20250116 02:59:30.628758 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.591183 22518 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.591212 23063 raft_consensus.cc:3049] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:30.524482 23109 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:30.564488 20504 rpcz_store.cc:269] 0116 02:59:12.151491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:12.160466 (+ 8975us) service_pool.cc:225] Handling call
0116 02:59:30.563935 (+18403469us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:22.910546 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:22.975726 (+ 65180us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:22.975777 (+ 51us) write_op.cc:435] Acquired schema lock
0116 02:59:22.975794 (+ 17us) tablet.cc:659] Decoding operations
0116 02:59:22.976264 (+ 470us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:22.976341 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:22.976367 (+ 26us) tablet.cc:682] Acquiring locks for 2 operations
0116 02:59:22.983864 (+ 7497us) tablet.cc:698] Row locks acquired
0116 02:59:22.983891 (+ 27us) write_op.cc:260] PREPARE: finished
0116 02:59:22.984093 (+ 202us) write_op.cc:270] Start()
0116 02:59:22.984242 (+ 149us) write_op.cc:276] Timestamp: P: 1736996362984008 usec, L: 0
0116 02:59:22.984262 (+ 20us) op_driver.cc:348] REPLICATION: starting
0116 02:59:22.985018 (+ 756us) log.cc:844] Serialized 930 byte log entry
0116 02:59:23.328375 (+343357us) op_driver.cc:464] REPLICATION: finished
0116 02:59:24.866937 (+1538562us) spinlock_profiling.cc:243] Waited 6.91 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc323db881e 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32a722d20
0116 02:59:25.707174 (+840237us) write_op.cc:301] APPLY: starting
0116 02:59:25.714921 (+ 7747us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:25.740944 (+ 26023us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:25.747247 (+ 6303us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:25.981925 (+234678us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:26.068589 (+ 86664us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0116 02:59:26.073853 (+ 5264us) write_op.cc:312] APPLY: finished
0116 02:59:26.139534 (+ 65681us) log.cc:844] Serialized 29 byte log entry
0116 02:59:26.431486 (+291952us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:26.435561 (+ 4075us) write_op.cc:454] Released schema lock
0116 02:59:26.450731 (+ 15170us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:26.821043 (+370312us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:26.846727 (+ 25684us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10211,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":2373566,"apply.run_cpu_time_us":14738,"apply.run_wall_time_us":812783,"num_ops":2,"prepare.queue_time_us":334186,"prepare.run_cpu_time_us":20860,"prepare.run_wall_time_us":184894,"raft.queue_time_us":118837,"raft.run_cpu_time_us":66007,"raft.run_wall_time_us":2417677,"replication_time_us":343933,"spinlock_wait_cycles":15887232,"thread_start_us":2941425,"threads_started":5,"wal-append.queue_time_us":480948,"wal-append.run_cpu_time_us":13565,"wal-append.run_wall_time_us":2479662}]]}
I20250116 02:59:30.616350 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.644995 20605 raft_consensus.cc:2405] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250116 02:59:30.620378 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1501) took 1863 ms. Trace:
I20250116 02:59:28.229121 20682 rpcz_store.cc:276] 0116 02:59:26.229029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.002375 (+1773346us) service_pool.cc:225] Handling call
0116 02:59:28.020905 (+ 18530us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:28.036060 (+ 15155us) raft_consensus.cc:1529] Early marking committed up to index 3
0116 02:59:28.073336 (+ 37276us) log.cc:844] Serialized 14 byte log entry
0116 02:59:28.222683 (+149347us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:28.222757 (+ 74us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:28.222834 (+ 77us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.222918 (+ 84us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:28.223087 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":43544,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":11609,"thread_start_us":123481,"threads_started":1}
I20250116 02:59:30.629168 20681 rpcz_store.cc:276] 0116 02:59:29.070970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.599056 (+1528086us) service_pool.cc:225] Handling call
0116 02:59:30.599385 (+ 329us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.612741 (+ 13356us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.612953 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.625314 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 374) took 1723 ms. Trace:
I20250116 02:59:30.610227 20751 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:30.650067 20678 rpcz_store.cc:276] 0116 02:59:28.902399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.591525 (+1689126us) service_pool.cc:225] Handling call
0116 02:59:30.624689 (+ 33164us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.624910 (+ 221us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.624974 (+ 64us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.625025 (+ 51us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:30.625070 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.625125 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.625244 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.619414 23025 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:30.661597 20681 raft_consensus.cc:1270] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250116 02:59:30.662418 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1578) took 1447 ms. Trace:
I20250116 02:59:30.662751 20681 rpcz_store.cc:276] 0116 02:59:29.214990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.660833 (+1445843us) service_pool.cc:225] Handling call
0116 02:59:30.661220 (+ 387us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.662129 (+ 909us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.662352 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.665211 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 376) took 1344 ms. Trace:
I20250116 02:59:30.599701 22924 consensus_queue.cc:1046] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.038s
I20250116 02:59:30.669921 20678 rpcz_store.cc:276] 0116 02:59:29.320566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.664355 (+1343789us) service_pool.cc:225] Handling call
0116 02:59:30.664672 (+ 317us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.664864 (+ 192us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.664889 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.664947 (+ 58us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:30.664990 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.665042 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.665147 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.636353 23063 raft_consensus.cc:738] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:30.672057 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.635501 23114 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:30.673524 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.671881 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.681761 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1575) took 1599 ms. Trace:
I20250116 02:59:30.682337 20682 rpcz_store.cc:276] 0116 02:59:29.082502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.646820 (+1564318us) service_pool.cc:225] Handling call
0116 02:59:30.681189 (+ 34369us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.681399 (+ 210us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.681425 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.681472 (+ 47us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:30.681505 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.681556 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.681691 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.675813 20601 raft_consensus.cc:1235] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250116 02:59:30.685170 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1650) took 1102 ms. Trace:
I20250116 02:59:30.685443 20601 rpcz_store.cc:276] 0116 02:59:29.583047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.636178 (+1053131us) service_pool.cc:225] Handling call
0116 02:59:30.675571 (+ 39393us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:30.684906 (+ 9335us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.685119 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.616142 23120 consensus_queue.cc:1046] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20250116 02:59:30.597991 22994 consensus_queue.cc:237] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.689069 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.679144 23113 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:30.693046 22980 consensus_meta.cc:287] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.503s user 0.008s sys 0.000s
I20250116 02:59:30.675215 23114 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.693542 22980 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:30.699756 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 173098us
I20250116 02:59:30.674579 20754 raft_consensus.cc:2405] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:30.666666 23025 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:30.636938 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:30.708485 20572 leader_election.cc:336] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.818s (SENT)
I20250116 02:59:30.709967 20572 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.707684 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 383) took 1482 ms. Trace:
I20250116 02:59:30.675689 23107 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.714430 20604 rpcz_store.cc:276] 0116 02:59:29.225461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.546879 (+1321418us) service_pool.cc:225] Handling call
0116 02:59:30.547221 (+ 342us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.547408 (+ 187us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:30.548001 (+ 593us) log.cc:844] Serialized 14 byte log entry
0116 02:59:30.707309 (+159308us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.707369 (+ 60us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:30.707418 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.707492 (+ 74us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.707623 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":155,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":3950,"thread_start_us":158819,"threads_started":1}
I20250116 02:59:30.732769 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:30.704286 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:30.709583 23061 consensus_meta.cc:287] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.722s user 0.012s sys 0.000s
I20250116 02:59:30.706451 22957 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.712034 23025 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:30.707878 23126 raft_consensus.cc:2743] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:30.768415 20678 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250116 02:59:30.626307 22791 consensus_queue.cc:260] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.737257 23108 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:30.734988 22715 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.701609 23094 consensus_queue.cc:1046] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20250116 02:59:30.785764 20604 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250116 02:59:30.773998 21331 raft_consensus.cc:2388] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:30.793831 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34864 (request call id 1167) took 2986 ms. Trace:
W20250116 02:59:30.801260 22846 consensus_meta.cc:287] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.828s user 0.010s sys 0.001s
I20250116 02:59:30.872764 22846 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:30.802433 23116 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:30.874666 20749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1936) took 2128 ms (client timeout 2591 ms). Trace:
I20250116 02:59:30.875083 22846 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.875805 23116 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.875697 20682 rpcz_store.cc:276] 0116 02:59:27.807423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.763841 (+2956418us) service_pool.cc:225] Handling call
0116 02:59:30.793732 (+ 29891us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:30.877291 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 426) took 2421 ms. Trace:
I20250116 02:59:30.803329 22980 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.803357 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.796530 23094 consensus_queue.cc:1225] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [LEADER]: Peer f65e8d7f76144fab9725ca32a6328c3f log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250116 02:59:30.878458 20754 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:30.814903 21334 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250116 02:59:30.879510 21334 pending_rounds.cc:77] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Aborting all ops after (but not including) 5
I20250116 02:59:30.879757 21334 pending_rounds.cc:99] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250116 02:59:30.879971 21334 raft_consensus.cc:2883] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:30.880764 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 412) took 4736 ms. Trace:
I20250116 02:59:30.881916 22868 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:30.844919 20720 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35540 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1526)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1505)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1503)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1544)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 443)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1540)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1924)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1928)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1939)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1940)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 439)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1536)
I20250116 02:59:30.882489 22868 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:30.816011 20572 leader_election.cc:336] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.879s (SENT)
I20250116 02:59:30.884372 23071 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.885885 20572 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:30.817229 23025 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.818390 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.765821 23061 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.821643 23089 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:30.789526 23108 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.822778 22927 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:30.816097 23072 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:30.830192 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:30.757404 23143 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.073s
I20250116 02:59:30.831079 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:30.835028 22872 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.849253 20571 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:30.814477 22821 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:30.861316 20674 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250116 02:59:30.868443 22843 consensus_meta.cc:287] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.703s user 0.010s sys 0.000s
I20250116 02:59:30.869199 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:30.884500 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:30.886868 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 71060.1us
I20250116 02:59:30.910465 20723 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:30.898260 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 49146.6us
I20250116 02:59:30.881199 20757 rpcz_store.cc:276] 0116 02:59:26.144612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:27.432183 (+1287571us) service_pool.cc:225] Handling call
0116 02:59:28.282217 (+850034us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:28.282461 (+ 244us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:28.282510 (+ 49us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:28.286219 (+ 3709us) log.cc:844] Serialized 26 byte log entry
0116 02:59:28.403198 (+116979us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:28.403237 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:28.403308 (+ 71us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:29.239961 (+836653us) raft_consensus.cc:1680] finished
0116 02:59:29.239986 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.880471 (+1640485us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b540012dc50. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:30.880696 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1936341248,"thread_start_us":46870,"threads_started":1,"wal-append.queue_time_us":150982,"wal-append.run_cpu_time_us":4014,"wal-append.run_wall_time_us":1756341}
I20250116 02:59:30.914494 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:30.900828 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 204449us
I20250116 02:59:30.907666 20750 raft_consensus.cc:2388] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:30.890100 20749 rpcz_store.cc:269] 0116 02:59:28.745894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.100356 (+354462us) service_pool.cc:225] Handling call
0116 02:59:30.874373 (+1774017us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b5400044554. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:30.874559 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":498236800}
I20250116 02:59:30.877712 20752 rpcz_store.cc:276] 0116 02:59:28.455984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.231716 (+775732us) service_pool.cc:225] Handling call
0116 02:59:30.875251 (+1643535us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.875483 (+ 232us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.875510 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.875558 (+ 48us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:30.875601 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.875661 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.877081 (+ 1420us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400044550. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:30.877219 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3742875392}
I20250116 02:59:30.894104 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:30.921607 20603 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:30.898844 20604 raft_consensus.cc:2388] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:30.907239 21332 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:30.935432 20757 raft_consensus.cc:2405] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:30.948546 20677 raft_consensus.cc:1235] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:30.952445 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:30.953284 20749 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 1.
I20250116 02:59:30.972330 20603 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:30.973626 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:30.974334 20603 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 5
W20250116 02:59:30.947090 23161 log.cc:926] Time spent T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
W20250116 02:59:30.906800 23145 log.cc:926] Time spent T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.136s user 0.001s sys 0.004s
I20250116 02:59:30.913429 20748 raft_consensus.cc:2363] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6.
W20250116 02:59:30.934391 23150 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.081s user 0.000s sys 0.003s
I20250116 02:59:30.886569 23071 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.645474 20683 rpcz_store.cc:276] 0116 02:59:28.757067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.297137 (+1540070us) service_pool.cc:225] Handling call
0116 02:59:30.320147 (+ 23010us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.320413 (+ 266us) raft_consensus.cc:1529] Early marking committed up to index 2
0116 02:59:30.338353 (+ 17940us) log.cc:844] Serialized 14 byte log entry
0116 02:59:30.620011 (+281658us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.620066 (+ 55us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:30.620129 (+ 63us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.620198 (+ 69us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:30.620318 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19772,"prepare.queue_time_us":19289,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":20434,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":19862,"thread_start_us":260966,"threads_started":1}
I20250116 02:59:31.139700 20678 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:30.888787 20602 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:31.013810 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.300135 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.037892 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:31.115464 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 171222us
W20250116 02:59:30.914248 23153 log.cc:926] Time spent T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.052s user 0.000s sys 0.004s
W20250116 02:59:30.978701 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 130878us
I20250116 02:59:31.013083 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:30.888415 23025 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:31.244637 20681 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.018s sys 0.022s
I20250116 02:59:30.949585 23107 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:30.887758 23109 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:30.881037 20675 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:31.307106 20649 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:30.886720 22825 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:30.906433 22846 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:31.331569 22846 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:31.332101 22846 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:31.332890 22927 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:31.037670 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.335466 20750 raft_consensus.cc:2371] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5. Re-sending same reply.
I20250116 02:59:31.336176 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1505) took 1313 ms. Trace:
I20250116 02:59:31.336566 20750 rpcz_store.cc:276] 0116 02:59:30.022854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.934414 (+911560us) service_pool.cc:225] Handling call
0116 02:59:31.336110 (+401696us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.338038 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:30.919605 23072 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.338982 23072 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:31.341297 20720 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50646 dropped due to backpressure: service queue is full with 50 items [suppressed 2 similar messages]
I20250116 02:59:31.342160 23072 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.343016 22821 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.328873 22519 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:31.344110 23054 consensus_meta.cc:287] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.865s user 0.008s sys 0.002s
I20250116 02:59:31.344002 22519 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.344570 23054 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:30.976953 23089 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:31.345538 23089 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.347652 20676 raft_consensus.cc:1235] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250116 02:59:31.310287 20752 raft_consensus.cc:2388] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:30.948812 23127 log.cc:926] Time spent T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 1.137s user 0.005s sys 0.009s
I20250116 02:59:31.099126 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 365) took 3300 ms. Trace:
I20250116 02:59:30.976665 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:31.354218 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1514) took 1328 ms. Trace:
W20250116 02:59:31.312239 20647 leader_election.cc:336] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.527s (SENT)
W20250116 02:59:30.977057 23152 log.cc:926] Time spent T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.126s user 0.001s sys 0.009s
I20250116 02:59:31.302817 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1515) took 1277 ms. Trace:
I20250116 02:59:31.254963 23061 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.309223 23108 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:30.913906 22843 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:31.118667 22936 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:31.305737 22885 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.029s sys 0.036s
I20250116 02:59:31.319322 23094 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:31.313361 22879 raft_consensus.cc:3049] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:31.322708 23071 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:31.146490 22924 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.324910 22920 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:31.328166 20603 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:31.367558 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:31.368436 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:31.329902 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.321236 22959 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.334529 22994 consensus_queue.cc:1046] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20250116 02:59:31.303992 20572 leader_election.cc:336] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.370s (SENT)
I20250116 02:59:31.347369 20756 raft_consensus.cc:2388] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:31.349334 20750 raft_consensus.cc:2371] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3. Re-sending same reply.
I20250116 02:59:31.349975 20646 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:31.341648 20720 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50646 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1544)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1545)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 443)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1547)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1546)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1553)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1540)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1924)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1928)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1939)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1940)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 439)
I20250116 02:59:31.351944 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:31.327842 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.352689 23144 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:31.353229 23054 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:31.379668 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 75709.8us
I20250116 02:59:31.369246 20683 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:31.330327 20749 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:31.381919 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1503) took 1359 ms. Trace:
I20250116 02:59:31.381544 20682 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:31.353169 20680 rpcz_store.cc:276] 0116 02:59:27.799291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.851942 (+3052651us) service_pool.cc:225] Handling call
0116 02:59:30.852310 (+ 368us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.852562 (+ 252us) raft_consensus.cc:1529] Early marking committed up to index 5
0116 02:59:30.852922 (+ 360us) log.cc:844] Serialized 14 byte log entry
0116 02:59:31.098662 (+245740us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:31.098761 (+ 99us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:31.098825 (+ 64us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:31.098900 (+ 75us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:31.099050 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":648,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":143,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":31507,"thread_start_us":245173,"threads_started":1}
I20250116 02:59:31.384425 20749 rpcz_store.cc:276] 0116 02:59:30.022349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.012792 (+990443us) service_pool.cc:225] Handling call
0116 02:59:31.381836 (+369044us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.385033 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:31.385926 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.358853 20757 rpcz_store.cc:276] 0116 02:59:30.025541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.037571 (+1012030us) service_pool.cc:225] Handling call
0116 02:59:31.302746 (+265175us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:31.316136 20681 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s user 0.019s sys 0.022s
I20250116 02:59:31.012199 22993 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:30.896238 23100 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.421s
I20250116 02:59:31.180593 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 779) took 2406 ms. Trace:
I20250116 02:59:31.014225 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1572) took 2665 ms. Trace:
W20250116 02:59:31.361613 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 59175.1us
I20250116 02:59:30.951757 22715 consensus_queue.cc:1035] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.396s
I20250116 02:59:31.012280 22949 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:30.955427 22957 consensus_queue.cc:1035] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.407s
I20250116 02:59:31.012710 23113 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:31.363817 22821 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:31.326975 23107 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:30.892520 23115 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.380447 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1516) took 1354 ms. Trace:
I20250116 02:59:31.437752 20750 rpcz_store.cc:276] 0116 02:59:30.025795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.337637 (+1311842us) service_pool.cc:225] Handling call
0116 02:59:31.380326 (+ 42689us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.380977 20603 raft_consensus.cc:2405] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250116 02:59:31.382221 20678 raft_consensus.cc:2363] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:31.358582 20675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1590) took 1294 ms. Trace:
I20250116 02:59:31.383167 20571 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:31.380587 20572 leader_election.cc:336] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.403s (SENT)
W20250116 02:59:31.364517 22885 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.650s user 0.029s sys 0.036s
I20250116 02:59:31.355496 20647 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:31.355186 20752 rpcz_store.cc:276] 0116 02:59:30.025300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.920830 (+895530us) service_pool.cc:225] Handling call
0116 02:59:31.354142 (+433312us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.387183 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.423627 21299 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:31.387266 23144 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.425781 21329 rpcz_store.cc:276] 0116 02:59:28.774337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:28.778570 (+ 4233us) service_pool.cc:225] Handling call
0116 02:59:30.537192 (+1758622us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98cbc 00007fc323d9887b 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:30.537468 (+ 276us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.537761 (+ 293us) raft_consensus.cc:1529] Early marking committed up to index 5
0116 02:59:30.761843 (+224082us) log.cc:844] Serialized 14 byte log entry
0116 02:59:30.950263 (+188420us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:30.951845 (+ 1582us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:31.012561 (+ 60716us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:31.048080 (+ 35519us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:31.147377 (+ 99297us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12811,"prepare.queue_time_us":244335,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":42953,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":28780,"spinlock_wait_cycles":29401984,"thread_start_us":332257,"threads_started":2,"wal-append.queue_time_us":124551}
I20250116 02:59:31.474982 23144 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.438838 22993 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.409547 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.476969 20683 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:31.477932 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.478786 20683 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:31.479682 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:31.480252 20683 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:31.481317 22993 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.428247 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:31.439739 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1579) took 2060 ms. Trace:
W20250116 02:59:31.441785 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 90945.6us
I20250116 02:59:31.440866 23054 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.486850 23054 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.487342 23115 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:31.487941 23115 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.443006 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 380) took 2666 ms. Trace:
I20250116 02:59:31.444365 22980 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:31.489310 20608 rpcz_store.cc:276] 0116 02:59:28.776600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.476706 (+1700106us) service_pool.cc:225] Handling call
0116 02:59:30.487076 (+ 10370us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:30.487557 (+ 481us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.487609 (+ 52us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:30.488124 (+ 515us) log.cc:844] Serialized 26 byte log entry
0116 02:59:30.887543 (+399419us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:30.887585 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.887782 (+ 197us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:31.442738 (+554956us) raft_consensus.cc:1680] finished
0116 02:59:31.442778 (+ 40us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:31.442939 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":302440,"threads_started":1,"wal-append.queue_time_us":353336}
I20250116 02:59:31.490784 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:31.491700 20753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1523) took 1464 ms (client timeout 1764 ms). Trace:
I20250116 02:59:31.492043 23089 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:31.492616 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 367) took 3510 ms. Trace:
I20250116 02:59:31.492733 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:31.408632 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.369204 23094 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.493594 22519 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.387467 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:31.383311 22924 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.493088 20679 rpcz_store.cc:276] 0116 02:59:27.982561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.246598 (+2264037us) service_pool.cc:225] Handling call
0116 02:59:30.247081 (+ 483us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:30.257392 (+ 10311us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.257428 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:30.267315 (+ 9887us) log.cc:844] Serialized 26 byte log entry
0116 02:59:30.894667 (+627352us) spinlock_profiling.cc:243] Waited 6.14 ms on lock 0x7b18002da3c0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32698be4b 00007fc326991913 00007fc326991cc1 00007fc326ce345f
0116 02:59:30.934583 (+ 39916us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:30.934759 (+ 176us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.934841 (+ 82us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:31.492326 (+557485us) raft_consensus.cc:1680] finished
0116 02:59:31.492350 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:31.492557 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1138,"raft.queue_time_us":6151,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7224,"spinlock_wait_cycles":14126848,"thread_start_us":39022,"threads_started":1,"wal-append.queue_time_us":1125285}
W20250116 02:59:31.492064 20753 rpcz_store.cc:269] 0116 02:59:30.027597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.490457 (+1462860us) service_pool.cc:225] Handling call
0116 02:59:31.491635 (+ 1178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.473281 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.443229 20572 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:31.476009 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.498371 20750 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:31.477993 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.367280 22879 raft_consensus.cc:738] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:31.488344 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1505) took 2704 ms. Trace:
W20250116 02:59:31.439771 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 56747us
I20250116 02:59:31.437428 23107 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:31.511201 20757 raft_consensus.cc:2363] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
W20250116 02:59:31.489197 22600 consensus_meta.cc:287] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.842s user 0.006s sys 0.006s
I20250116 02:59:31.520936 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.523381 20571 leader_election.cc:400] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:31.523757 20571 leader_election.cc:403] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250116 02:59:31.519812 20676 rpcz_store.cc:276] 0116 02:59:28.784155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.586445 (+1802290us) service_pool.cc:225] Handling call
0116 02:59:31.347396 (+760951us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:31.348087 (+ 691us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:31.488128 (+140041us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b5400323a50. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:31.488278 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1716762880}
I20250116 02:59:31.524827 20571 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:31.524153 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1518) took 1497 ms. Trace:
I20250116 02:59:31.525563 20757 rpcz_store.cc:276] 0116 02:59:30.026299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.386864 (+1360565us) service_pool.cc:225] Handling call
0116 02:59:31.524082 (+137218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.526525 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.527292 20757 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:31.529229 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1499) took 1507 ms. Trace:
I20250116 02:59:31.529387 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.530222 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.496448 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.531211 20749 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:31.531445 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 107924us
W20250116 02:59:31.532279 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 220072us
W20250116 02:59:31.533078 20647 leader_election.cc:336] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.591s (SENT)
I20250116 02:59:31.533625 20647 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:31.409305 20649 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:31.529618 20757 rpcz_store.cc:276] 0116 02:59:30.021391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.526238 (+1504847us) service_pool.cc:225] Handling call
0116 02:59:31.529172 (+ 2934us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.535117 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.535815 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.536242 20571 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:31.536446 20757 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:31.537122 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1529) took 1483 ms. Trace:
W20250116 02:59:31.536357 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 127155us
I20250116 02:59:31.537493 20757 rpcz_store.cc:276] 0116 02:59:30.053743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.535504 (+1481761us) service_pool.cc:225] Handling call
0116 02:59:31.537068 (+ 1564us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.538293 20649 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:31.538810 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.489863 22980 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:31.530073 20753 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:31.444623 23130 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:31.540309 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1535) took 1153 ms. Trace:
I20250116 02:59:31.545069 20753 rpcz_store.cc:276] 0116 02:59:30.386480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.497188 (+1110708us) service_pool.cc:225] Handling call
0116 02:59:31.540247 (+ 43059us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.531008 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1528) took 1477 ms. Trace:
I20250116 02:59:31.541270 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.426492 22949 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.542903 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.543251 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.519527 23071 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:31.547237 20752 rpcz_store.cc:276] 0116 02:59:30.053441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.511812 (+1458371us) service_pool.cc:225] Handling call
0116 02:59:31.530945 (+ 19133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.519670 22994 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:31.551337 21332 raft_consensus.cc:2405] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250116 02:59:31.562529 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.519996 23114 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:31.550161 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1496) took 1529 ms. Trace:
I20250116 02:59:31.556141 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.557875 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:31.556161 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.434273 23113 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.563959 20749 raft_consensus.cc:2371] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5. Re-sending same reply.
I20250116 02:59:31.566018 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:31.566181 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1532) took 1181 ms. Trace:
I20250116 02:59:31.563486 20750 rpcz_store.cc:276] 0116 02:59:30.020665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.499066 (+1478401us) service_pool.cc:225] Handling call
0116 02:59:31.550096 (+ 51030us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.566901 20749 rpcz_store.cc:276] 0116 02:59:30.385214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.545205 (+1159991us) service_pool.cc:225] Handling call
0116 02:59:31.566117 (+ 20912us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.523488 21329 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:31.543747 22843 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:31.572847 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.544481 22980 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:31.573616 20750 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:31.539705 22715 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:31.544689 22519 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:31.569240 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:31.570263 21302 leader_election.cc:336] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.553s (SENT)
I20250116 02:59:31.545014 23130 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:31.540911 22872 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:31.574697 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:31.576251 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:31.593780 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.576304 21300 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:31.577039 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.583843 21302 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:31.595877 20752 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:31.589998 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:31.562641 22994 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:31.593667 20753 raft_consensus.cc:2371] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4. Re-sending same reply.
I20250116 02:59:31.616681 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1533) took 1231 ms. Trace:
I20250116 02:59:31.617094 20753 rpcz_store.cc:276] 0116 02:59:30.385600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.545927 (+1160327us) service_pool.cc:225] Handling call
0116 02:59:31.616593 (+ 70666us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.597791 21336 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:31.612596 22519 raft_consensus.cc:695] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:31.589119 23061 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.540009 22600 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.620998 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.628430 20752 raft_consensus.cc:2388] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:31.622529 21335 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:31.589082 23114 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:31.597519 23130 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:31.577116 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:31.624863 23094 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.627264 23100 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:31.638103 22821 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:31.625962 23061 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.635344 23054 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.596985 22846 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.643177 22924 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:31.627211 22993 raft_consensus.cc:2798] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:31.647511 22600 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.627235 22994 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:31.655887 23120 raft_consensus.cc:3049] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:31.656368 23120 raft_consensus.cc:738] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:31.625881 22519 consensus_queue.cc:237] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.657274 23120 consensus_queue.cc:260] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:31.658321 23120 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:31.660185 23063 consensus_queue.cc:260] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.661690 23063 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:31.664844 23111 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.637485 23114 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:31.576906 22843 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:31.589082 23171 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.625552 22949 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.621446 23115 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.652055 22993 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:31.665023 23154 log.cc:926] Time spent T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.876s user 0.003s sys 0.003s
I20250116 02:59:31.665417 23111 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.668557 23114 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:31.671705 23171 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.693557 23171 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:31.699692 23130 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.732417 20602 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:31.484509 20601 raft_consensus.cc:1235] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250116 02:59:31.736662 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1654) took 1690 ms. Trace:
I20250116 02:59:31.737221 20601 rpcz_store.cc:276] 0116 02:59:30.046344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.686135 (+639791us) service_pool.cc:225] Handling call
0116 02:59:31.484008 (+797873us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:31.484227 (+ 219us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:31.735743 (+251516us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:31.736434 (+ 691us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13409152}
I20250116 02:59:31.740132 20601 raft_consensus.cc:1270] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250116 02:59:31.742048 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 361353us
I20250116 02:59:31.750159 20605 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250116 02:59:31.751125 20571 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:31.408946 20674 rpcz_store.cc:276] 0116 02:59:28.326546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.239984 (+1913438us) service_pool.cc:225] Handling call
0116 02:59:30.860537 (+620553us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:30.860838 (+ 301us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.906312 (+ 45474us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.992122 (+ 85810us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7572736}
I20250116 02:59:31.757408 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.760239 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.761878 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.763433 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.764889 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.441377 20675 rpcz_store.cc:276] 0116 02:59:30.064355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.161227 (+ 96872us) service_pool.cc:225] Handling call
0116 02:59:30.551921 (+390694us) spinlock_profiling.cc:243] Waited 7.31 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98cbc 00007fc323d9887b 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:31.352860 (+800939us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16824192}
I20250116 02:59:31.768323 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.769850 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:31.771277 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:31.772491 20601 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:31.777209 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:31.777678 20675 raft_consensus.cc:1270] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:31.575348 20680 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:31.798455 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.801194 20679 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:31.807314 20682 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:31.808401 20572 leader_election.cc:336] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.191s (SENT)
W20250116 02:59:31.816346 20720 connection.cc:232] RPC call timeout handler was delayed by -1.254484214999934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
W20250116 02:59:31.817215 20720 connection.cc:232] RPC call timeout handler was delayed by -1.0105842149998807s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20250116 02:59:31.816485 20572 leader_election.cc:336] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.196s (SENT)
I20250116 02:59:31.833731 20749 raft_consensus.cc:2388] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:31.626284 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.839334 20750 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:31.839982 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1538) took 1215 ms. Trace:
I20250116 02:59:31.840338 20750 rpcz_store.cc:276] 0116 02:59:30.624233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.612881 (+988648us) service_pool.cc:225] Handling call
0116 02:59:31.839897 (+227016us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.835227 20755 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:31.835856 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1543) took 1062 ms. Trace:
I20250116 02:59:31.843327 20752 rpcz_store.cc:276] 0116 02:59:30.774347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.615290 (+840943us) service_pool.cc:225] Handling call
0116 02:59:31.835527 (+220237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.837342 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:31.845086 20753 raft_consensus.cc:2388] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:31.845660 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1519) took 1819 ms. Trace:
I20250116 02:59:31.846036 20753 rpcz_store.cc:276] 0116 02:59:30.026543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.617787 (+1591244us) service_pool.cc:225] Handling call
0116 02:59:31.845570 (+227783us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.618446 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.857726 21302 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:31.640293 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:31.650538 21331 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:31.861702 22920 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:31.862107 22884 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:31.863163 22884 raft_consensus.cc:695] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:31.862679 22920 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.864874 22884 consensus_queue.cc:237] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.866362 22920 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.867329 22825 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:31.867974 22924 raft_consensus.cc:3049] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:31.868432 22924 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:31.667511 22843 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:31.870558 22715 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:31.869891 22924 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.871481 22924 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:31.871701 22715 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:31.872541 22715 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:31.873142 22715 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.867358 22920 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.876852 22920 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.878041 22846 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.573308 22879 consensus_queue.cc:260] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.879758 22879 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:31.880365 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.881461 21335 raft_consensus.cc:2388] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:31.884102 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.886127 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.888048 22821 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.869824 22600 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.889066 22920 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:31.888828 22821 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.889431 22600 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.868388 22825 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:31.895915 22600 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.896629 22600 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:31.887441 22864 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.897441 22600 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.897588 22864 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.900475 22864 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.901400 22600 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.902136 22715 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:31.766382 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.904469 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.904750 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:31.906193 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.906507 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.673319 22993 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:31.907747 22927 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:31.898192 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.910800 22821 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.915632 23054 consensus_queue.cc:1046] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:31.915920 22994 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.845786 20601 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:31.924044 22980 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.842363 20683 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:31.953107 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:32.013497 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 390) took 1452 ms. Trace:
I20250116 02:59:31.950336 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:31.896600 22825 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:31.895924 22920 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:31.916980 23061 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:32.028455 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1597) took 1067 ms. Trace:
I20250116 02:59:32.035313 20683 rpcz_store.cc:276] 0116 02:59:30.957131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.479407 (+522276us) service_pool.cc:225] Handling call
0116 02:59:32.021650 (+542243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.995554 22884 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:32.042935 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:31.702371 23130 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:32.045531 21336 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
W20250116 02:59:32.048666 20607 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 386) took 1877 ms (client timeout 1553 ms). Trace:
I20250116 02:59:32.054392 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:32.061097 20602 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
W20250116 02:59:32.061323 20754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1511) took 2036 ms (client timeout 2005 ms). Trace:
W20250116 02:59:32.063788 20756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1500) took 2042 ms (client timeout 1947 ms). Trace:
I20250116 02:59:32.063557 22884 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:31.997212 23071 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:32.071843 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:31.689988 23113 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:32.074976 20601 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:31.992332 23083 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:32.092058 22879 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:32.097594 22879 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.099609 20674 raft_consensus.cc:2388] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:32.100901 22879 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:32.101769 21331 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:32.102924 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:32.103734 20601 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:32.013990 20604 rpcz_store.cc:276] 0116 02:59:30.560681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.368612 (+807931us) service_pool.cc:225] Handling call
0116 02:59:31.408851 (+ 40239us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:31.409059 (+ 208us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:31.743375 (+334316us) log.cc:844] Serialized 14 byte log entry
0116 02:59:32.012676 (+269301us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.012800 (+ 124us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:32.012940 (+ 140us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.013035 (+ 95us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.013275 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43007,"prepare.queue_time_us":268313,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":429,"raft.queue_time_us":59409,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":217,"thread_start_us":520775,"threads_started":2}
W20250116 02:59:32.075753 20607 rpcz_store.cc:269] 0116 02:59:30.171384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.251500 (+ 80116us) service_pool.cc:225] Handling call
0116 02:59:32.048148 (+1796648us) spinlock_profiling.cc:243] Waited 91.7 ms on lock 0x7b540014e1d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:32.048442 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":210973568}
W20250116 02:59:32.090075 20756 rpcz_store.cc:269] 0116 02:59:30.021630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.593020 (+571390us) service_pool.cc:225] Handling call
0116 02:59:32.063251 (+1470231us) spinlock_profiling.cc:243] Waited 751 ms on lock 0x7b540012dc54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:32.063515 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1726435072}
W20250116 02:59:32.116216 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 410912us
I20250116 02:59:32.116253 22868 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:32.117364 20649 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:32.119372 23140 log.cc:926] Time spent T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 1.784s user 0.014s sys 0.012s
I20250116 02:59:32.119871 22980 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:32.123188 22884 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:32.123812 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1901) took 3631 ms. Trace:
I20250116 02:59:32.124404 20755 rpcz_store.cc:276] 0116 02:59:28.492203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.907876 (+1415673us) service_pool.cc:225] Handling call
0116 02:59:29.908201 (+ 325us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:31.842813 (+1934612us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.123443 (+280630us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b5400261454. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:32.123702 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4429491584}
I20250116 02:59:32.125380 22843 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.117715 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:32.126713 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1544) took 1282 ms. Trace:
I20250116 02:59:32.111248 22920 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.133327 20571 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:32.133904 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 376) took 4131 ms. Trace:
I20250116 02:59:32.127619 20749 rpcz_store.cc:276] 0116 02:59:30.844182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.567739 (+723557us) service_pool.cc:225] Handling call
0116 02:59:32.126403 (+558664us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b5400043b54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:32.126595 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":573872512}
I20250116 02:59:32.143785 22920 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:32.163005 20680 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
W20250116 02:59:32.124653 20647 leader_election.cc:336] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.104s (SENT)
I20250116 02:59:32.143997 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1660) took 1589 ms. Trace:
I20250116 02:59:32.137126 22927 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.145407 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:32.146945 22959 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:32.148029 20649 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:31.995209 23115 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.133s
I20250116 02:59:32.151087 23113 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election won for term 5
W20250116 02:59:32.152238 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 196826us
I20250116 02:59:32.151794 23071 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.156303 22843 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:32.157503 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:32.160269 20603 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.010s sys 0.032s
I20250116 02:59:32.159921 22936 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:31.826234 20720 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50656 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 443)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1547)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1546)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1553)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1947)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1949)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1555)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1944)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1950)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1946)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1954)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1951)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1952)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1956)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1948)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1943)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1924)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1928)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1939)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1940)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 439)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 447)
I20250116 02:59:32.174089 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:32.264354 20606 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1662) took 1709 ms (client timeout 2019 ms). Trace:
W20250116 02:59:32.264887 20606 rpcz_store.cc:269] 0116 02:59:30.555175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.564180 (+ 9005us) service_pool.cc:225] Handling call
0116 02:59:32.173723 (+1609543us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:32.264262 (+ 90539us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64584448}
I20250116 02:59:32.266038 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.230240 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:32.231062 22908 raft_consensus.cc:2743] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:32.173141 22994 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:32.171770 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1503) took 3408 ms. Trace:
I20250116 02:59:32.281199 20677 rpcz_store.cc:276] 0116 02:59:28.763633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.584297 (+1820664us) service_pool.cc:225] Handling call
0116 02:59:30.584618 (+ 321us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.949000 (+364382us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.171263 (+1222263us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54000875d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:32.171576 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":836466048}
I20250116 02:59:32.282006 23071 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:32.245486 23114 consensus_meta.cc:287] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.565s user 0.011s sys 0.000s
I20250116 02:59:32.166972 23111 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:32.273288 20682 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:32.283593 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.282757 23114 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.284396 20604 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:32.285151 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:32.230644 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 516) took 1992 ms. Trace:
W20250116 02:59:32.272681 20603 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s user 0.013s sys 0.033s
I20250116 02:59:32.289721 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:32.290201 22957 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:32.289601 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 391) took 1727 ms. Trace:
I20250116 02:59:32.291148 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.291247 20603 rpcz_store.cc:276] 0116 02:59:30.562023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.441858 (+879835us) service_pool.cc:225] Handling call
0116 02:59:31.442242 (+ 384us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:31.442633 (+ 391us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:31.442665 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:31.471469 (+ 28804us) log.cc:844] Serialized 26 byte log entry
0116 02:59:32.289149 (+817680us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:32.289212 (+ 63us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.289322 (+ 110us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:32.289383 (+ 61us) raft_consensus.cc:1680] finished
0116 02:59:32.289401 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.289547 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2460,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":145,"thread_start_us":795810,"threads_started":1,"wal-append.queue_time_us":569302}
I20250116 02:59:32.289479 23113 raft_consensus.cc:695] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:32.296123 23113 consensus_queue.cc:237] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.294965 21328 rpcz_store.cc:276] 0116 02:59:30.237794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.241146 (+ 3352us) service_pool.cc:225] Handling call
0116 02:59:30.247082 (+ 5936us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:30.247588 (+ 506us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.247626 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:30.318978 (+ 71352us) log.cc:844] Serialized 52 byte log entry
0116 02:59:30.628928 (+309950us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:30.977781 (+348853us) log.cc:844] Serialized 14 byte log entry
0116 02:59:32.178418 (+1200637us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b18003cf6a0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323da3ec6 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326ca3213 00007fc326d0ff0f 00007fc326d0e99e
0116 02:59:32.229628 (+ 51210us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b5400470520. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326ca3213 00007fc326d0ff0f 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc
0116 02:59:32.229734 (+ 106us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.229945 (+ 211us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:32.230044 (+ 99us) raft_consensus.cc:1680] finished
0116 02:59:32.230064 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.230366 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":102164,"prepare.queue_time_us":185201,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":28,"raft.queue_time_us":84616,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":45234,"spinlock_wait_cycles":99899392,"thread_start_us":452745,"threads_started":2,"wal-append.queue_time_us":431757}
I20250116 02:59:32.297458 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:32.294785 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2031) took 1534 ms. Trace:
I20250116 02:59:32.238224 20601 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:32.299062 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:32.300009 20677 raft_consensus.cc:2363] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:32.300218 22927 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:32.300854 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:32.175210 22884 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:32.298424 20603 rpcz_store.cc:276] 0116 02:59:30.760451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.293728 (+1533277us) service_pool.cc:225] Handling call
0116 02:59:32.294179 (+ 451us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.294411 (+ 232us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.294437 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.294477 (+ 40us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:32.294510 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.294568 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.294705 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:32.259595 20647 connection.cc:232] RPC call timeout handler was delayed by -1.0498525589999872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1874s for a response
I20250116 02:59:32.173435 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 387) took 1217 ms. Trace:
W20250116 02:59:32.089708 20754 rpcz_store.cc:269] 0116 02:59:30.024566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.060890 (+2036324us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:32.061089 (+ 199us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:32.275343 20752 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:32.280228 20607 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:32.215837 23126 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:32.134358 20600 rpcz_store.cc:276] 0116 02:59:28.002321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:29.981088 (+1978767us) service_pool.cc:225] Handling call
0116 02:59:29.981491 (+ 403us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:30.005713 (+ 24222us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:30.007343 (+ 1630us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:30.053066 (+ 45723us) log.cc:844] Serialized 26 byte log entry
0116 02:59:30.244951 (+191885us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:30.245003 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:30.245078 (+ 75us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:32.118361 (+1873283us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:32.133298 (+ 14937us) raft_consensus.cc:1680] finished
0116 02:59:32.133320 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.133692 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5148,"raft.queue_time_us":21802,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":24733,"spinlock_wait_cycles":8293120,"thread_start_us":181914,"threads_started":1,"wal-append.queue_time_us":207314}
I20250116 02:59:32.238396 23083 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.285369 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.274387 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:32.286407 20675 raft_consensus.cc:2405] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250116 02:59:32.287992 23114 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:32.128787 22825 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:32.301484 20677 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:32.271332 22791 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
W20250116 02:59:32.323019 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1502) took 2298 ms (client timeout 2894 ms). Trace:
W20250116 02:59:32.328058 20752 rpcz_store.cc:269] 0116 02:59:30.022118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.265653 (+2243535us) service_pool.cc:225] Handling call
0116 02:59:32.320215 (+ 54562us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:31.482964 20681 rpcz_store.cc:276] 0116 02:59:29.379905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.663525 (+1283620us) service_pool.cc:225] Handling call
0116 02:59:30.663835 (+ 310us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:30.664075 (+ 240us) raft_consensus.cc:1529] Early marking committed up to index 4
0116 02:59:30.713199 (+ 49124us) log.cc:844] Serialized 14 byte log entry
0116 02:59:31.439317 (+726118us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:31.439384 (+ 67us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:31.439445 (+ 61us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:31.439525 (+ 80us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:31.439668 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28799,"prepare.queue_time_us":48924,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":29084,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":30616,"spinlock_wait_cycles":1691392,"thread_start_us":695886,"threads_started":1,"wal-append.queue_time_us":679772}
I20250116 02:59:32.329923 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:32.331326 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:32.332008 20681 raft_consensus.cc:2371] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4. Re-sending same reply.
I20250116 02:59:32.332891 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:32.333472 20681 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:32.334354 22868 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:32.335678 20750 raft_consensus.cc:2371] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5. Re-sending same reply.
I20250116 02:59:32.335412 22868 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.338990 20604 raft_consensus.cc:2405] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:32.339563 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 394) took 1576 ms. Trace:
I20250116 02:59:32.339978 20603 rpcz_store.cc:276] 0116 02:59:30.762939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.321946 (+1559007us) service_pool.cc:225] Handling call
0116 02:59:32.338865 (+ 16919us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.339212 (+ 347us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.339236 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.339277 (+ 41us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:32.339316 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.339369 (+ 53us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.339482 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.340698 22868 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:32.341745 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2036) took 1447 ms. Trace:
I20250116 02:59:32.340742 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 395) took 1566 ms. Trace:
W20250116 02:59:32.173352 20748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1508) took 2149 ms (client timeout 1952 ms). Trace:
I20250116 02:59:32.342021 23083 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:32.279397 20599 rpcz_store.cc:276] 0116 02:59:30.554673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.631435 (+ 76762us) service_pool.cc:225] Handling call
0116 02:59:32.117371 (+1485936us) spinlock_profiling.cc:243] Waited 9.73 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:32.143922 (+ 26551us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22377856}
I20250116 02:59:32.343561 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2037) took 1448 ms. Trace:
I20250116 02:59:32.342660 20604 rpcz_store.cc:276] 0116 02:59:30.774346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.339770 (+1565424us) service_pool.cc:225] Handling call
0116 02:59:32.340158 (+ 388us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.340425 (+ 267us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.340447 (+ 22us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.340484 (+ 37us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:32.340522 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.340575 (+ 53us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.340664 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.344249 20600 rpcz_store.cc:276] 0116 02:59:30.894985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.342434 (+1447449us) service_pool.cc:225] Handling call
0116 02:59:32.342941 (+ 507us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.343185 (+ 244us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.343218 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.343263 (+ 45us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:32.343311 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.343370 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.343495 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.345813 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2038) took 1450 ms. Trace:
I20250116 02:59:32.346213 20604 rpcz_store.cc:276] 0116 02:59:30.895432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.344704 (+1449272us) service_pool.cc:225] Handling call
0116 02:59:32.345147 (+ 443us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.345424 (+ 277us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.345454 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.345523 (+ 69us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:32.345571 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.345636 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.345748 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:32.347234 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 199349us
W20250116 02:59:32.170030 23107 consensus_meta.cc:287] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.626s user 0.013s sys 0.002s
I20250116 02:59:32.178819 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:32.297595 20720 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50656 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1546)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1553)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1947)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1949)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1555)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1944)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1950)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1946)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1954)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1958)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1951)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1952)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1956)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1960)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1948)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1957)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1962)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1943)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1924)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1928)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1939)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1940)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 439)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 448)
I20250116 02:59:32.302867 20676 rpcz_store.cc:276] 0116 02:59:30.955436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.524899 (+569463us) service_pool.cc:225] Handling call
0116 02:59:31.546585 (+ 21686us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:31.546853 (+ 268us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:31.546876 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:31.547353 (+ 477us) log.cc:844] Serialized 26 byte log entry
0116 02:59:32.093309 (+545956us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:32.105496 (+ 12187us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.131109 (+ 25613us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:32.172993 (+ 41884us) raft_consensus.cc:1680] finished
0116 02:59:32.173019 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.173218 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":450540,"threads_started":1,"wal-append.queue_time_us":566190}
I20250116 02:59:32.342170 20603 rpcz_store.cc:276] 0116 02:59:30.894745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.340910 (+1446165us) service_pool.cc:225] Handling call
0116 02:59:32.341242 (+ 332us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.341442 (+ 200us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.341465 (+ 23us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.341502 (+ 37us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:32.341539 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.341590 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.341680 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.171800 22993 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:32.170560 23183 log.cc:926] Time spent T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20250116 02:59:32.363556 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2041) took 1059 ms. Trace:
I20250116 02:59:32.363951 20604 rpcz_store.cc:276] 0116 02:59:31.300570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.347220 (+1046650us) service_pool.cc:225] Handling call
0116 02:59:32.347628 (+ 408us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.359686 (+ 12058us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.359724 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.359772 (+ 48us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:32.359816 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.359889 (+ 73us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.360041 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:32.365092 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 50121.1us
I20250116 02:59:32.365509 20604 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:32.365994 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:32.366902 22871 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:32.365839 20572 leader_election.cc:336] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.753s (SENT)
I20250116 02:59:32.353525 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2039) took 1442 ms. Trace:
I20250116 02:59:32.367511 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:32.342926 20748 rpcz_store.cc:269] 0116 02:59:30.023761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.656054 (+632293us) service_pool.cc:225] Handling call
0116 02:59:32.172874 (+1516820us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540033c7d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:32.173129 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":310313344}
I20250116 02:59:32.375159 22980 consensus_queue.cc:1035] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:59:32.375591 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:32.376305 23083 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:32.377348 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.378088 20748 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:32.378511 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2042) took 1076 ms. Trace:
I20250116 02:59:32.379010 20599 rpcz_store.cc:276] 0116 02:59:31.301807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.362630 (+1060823us) service_pool.cc:225] Handling call
0116 02:59:32.377858 (+ 15228us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.378112 (+ 254us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.378153 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.378215 (+ 62us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:32.378275 (+ 60us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.378335 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.378452 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.377144 23083 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.380431 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:32.381181 20599 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:32.382225 23083 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:32.371481 20600 rpcz_store.cc:276] 0116 02:59:30.907274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.345533 (+1438259us) service_pool.cc:225] Handling call
0116 02:59:32.346015 (+ 482us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.346258 (+ 243us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.346284 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.346348 (+ 64us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:32.346439 (+ 91us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.346497 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.349423 (+ 2926us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.388954 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1652) took 2805 ms. Trace:
I20250116 02:59:32.389384 20605 rpcz_store.cc:276] 0116 02:59:29.583746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:30.668606 (+1084860us) service_pool.cc:225] Handling call
0116 02:59:30.668959 (+ 353us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:31.750899 (+1081940us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.388628 (+637729us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540017fcd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:32.388890 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2477241600}
I20250116 02:59:32.391170 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.396972 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 2040) took 1462 ms. Trace:
I20250116 02:59:32.397750 20723 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:32.397948 23171 raft_consensus.cc:3049] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:32.398779 23171 raft_consensus.cc:738] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:32.399371 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 386) took 1409 ms. Trace:
I20250116 02:59:32.347391 23126 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.395345 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:32.367957 22825 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.359167 23107 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.465294 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.410523 20603 raft_consensus.cc:1235] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:32.397439 20602 rpcz_store.cc:276] 0116 02:59:30.934308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.347904 (+413596us) service_pool.cc:225] Handling call
0116 02:59:32.396754 (+1048850us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b5400093154. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:32.396905 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":545571712}
I20250116 02:59:32.401405 20674 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:32.478282 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.370767 22884 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.475790 20752 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:32.482930 22914 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:32.463601 20751 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:32.494439 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1495) took 2474 ms. Trace:
I20250116 02:59:32.402446 20571 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:32.399792 20678 rpcz_store.cc:276] 0116 02:59:30.955201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.498053 (+542852us) service_pool.cc:225] Handling call
0116 02:59:31.529312 (+ 31259us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:31.529526 (+ 214us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:32.062743 (+533217us) log.cc:844] Serialized 14 byte log entry
0116 02:59:32.337295 (+274552us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.352663 (+ 15368us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:32.352745 (+ 82us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.359037 (+ 6292us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.364281 (+ 5244us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":120296,"prepare.queue_time_us":727285,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":165783,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":15748,"thread_start_us":544548,"threads_started":2,"wal-append.queue_time_us":227792}
I20250116 02:59:32.496675 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2043) took 1189 ms. Trace:
I20250116 02:59:32.485215 23126 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:32.423377 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 95851.7us
I20250116 02:59:32.442152 20605 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:32.389379 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.495723 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1546) took 1314 ms. Trace:
I20250116 02:59:32.503058 20750 rpcz_store.cc:276] 0116 02:59:31.181463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.477894 (+1296431us) service_pool.cc:225] Handling call
0116 02:59:32.495627 (+ 17733us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.497793 20603 rpcz_store.cc:276] 0116 02:59:31.302089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.362816 (+1060727us) service_pool.cc:225] Handling call
0116 02:59:32.410329 (+ 47513us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.491557 (+ 81228us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.491822 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.497648 20752 rpcz_store.cc:276] 0116 02:59:30.020421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.461756 (+2441335us) service_pool.cc:225] Handling call
0116 02:59:32.494307 (+ 32551us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.483681 20676 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20250116 02:59:32.489671 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 46634.2us
I20250116 02:59:32.505544 20757 raft_consensus.cc:2363] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:32.483551 22519 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:32.399765 22993 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:32.519986 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.512428 21333 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:32.505847 21302 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 02:59:32.524075 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 155904us
I20250116 02:59:32.520346 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:32.500715 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.500995 23083 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:32.535212 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.535434 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.527398 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.537472 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.543841 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1548) took 1214 ms. Trace:
I20250116 02:59:32.526996 23126 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:32.551424 20750 raft_consensus.cc:2388] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:32.543962 20720 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35540 dropped due to backpressure: service queue is full with 50 items [suppressed 2 similar messages]
I20250116 02:59:32.569805 20756 rpcz_store.cc:276] 0116 02:59:31.329450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.500336 (+1170886us) service_pool.cc:225] Handling call
0116 02:59:32.543775 (+ 43439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.527695 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.582177 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:32.552423 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 50035.8us
I20250116 02:59:32.399744 23171 consensus_queue.cc:260] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.585742 20605 raft_consensus.cc:2388] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:32.572758 20720 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35540 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1555)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1944)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1950)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1946)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1954)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1958)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1951)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1963)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1952)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1956)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1965)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1960)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1964)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1948)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1957)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1962)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1966)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1943)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1967)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1924)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1928)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1939)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1940)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 439)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 448)
W20250116 02:59:32.585319 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 56971.5us
I20250116 02:59:32.589926 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2033) took 1822 ms. Trace:
I20250116 02:59:32.505803 20678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34864
I20250116 02:59:32.590857 20601 rpcz_store.cc:276] 0116 02:59:30.760950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.299800 (+1538850us) service_pool.cc:225] Handling call
0116 02:59:32.300313 (+ 513us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.300644 (+ 331us) raft_consensus.cc:1529] Early marking committed up to index 5
0116 02:59:32.356203 (+ 55559us) log.cc:844] Serialized 14 byte log entry
0116 02:59:32.582838 (+226635us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.582909 (+ 71us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:32.582971 (+ 62us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.583052 (+ 81us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.583202 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":87351,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"thread_start_us":196901,"threads_started":2}
I20250116 02:59:32.542896 22936 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:32.593674 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1552) took 1247 ms. Trace:
I20250116 02:59:32.596197 20752 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:32.520087 22519 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.592872 20678 raft_consensus.cc:604] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Received request to transfer leadership
I20250116 02:59:32.542558 23083 raft_consensus.cc:695] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:32.599299 20750 rpcz_store.cc:276] 0116 02:59:31.340587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.503819 (+1163232us) service_pool.cc:225] Handling call
0116 02:59:32.587496 (+ 83677us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.540401 20600 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:32.542850 20748 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:32.603494 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 97740.5us
W20250116 02:59:32.605566 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 68289.4us
I20250116 02:59:32.607239 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.607556 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34864 (request call id 1169) took 1413 ms. Trace:
I20250116 02:59:32.618103 20605 raft_consensus.cc:2388] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:32.619665 20678 rpcz_store.cc:276] 0116 02:59:31.188402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.505510 (+1317108us) service_pool.cc:225] Handling call
0116 02:59:32.601517 (+ 96007us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.642554 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.598594 22936 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:32.655259 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.652199 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:32.659217 20720 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:32.640647 20721 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35540 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1950)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1946)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1954)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1958)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1951)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1963)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1952)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1956)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1965)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1960)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1964)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1948)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1957)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1962)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1966)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1943)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1967)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1924)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1928)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1939)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1940)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 439)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 448)
I20250116 02:59:32.533277 22914 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:32.701167 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 56207.9us
I20250116 02:59:32.672811 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:32.745625 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 52565.3us
W20250116 02:59:32.715564 20680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 385) took 1749 ms (client timeout 2197 ms). Trace:
I20250116 02:59:32.749806 20649 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:32.704519 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 60825.4us
W20250116 02:59:32.768685 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 126141us
I20250116 02:59:32.674505 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.751140 23107 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:32.675395 23083 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.779819 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2026) took 2020 ms. Trace:
I20250116 02:59:32.780299 20608 rpcz_store.cc:276] 0116 02:59:30.759085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.522728 (+763643us) service_pool.cc:225] Handling call
0116 02:59:32.057811 (+535083us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:32.058199 (+ 388us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.287033 (+228834us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:32.324351 (+ 37318us) log.cc:844] Serialized 14 byte log entry
0116 02:59:32.474348 (+149997us) log.cc:844] Serialized 14 byte log entry
0116 02:59:32.474714 (+ 366us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.474798 (+ 84us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:32.474875 (+ 77us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.779425 (+304550us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54002d8a54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:32.779483 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:32.779618 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":75909,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":82,"raft.queue_time_us":223,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":10219,"spinlock_wait_cycles":510997120,"thread_start_us":144560,"threads_started":2,"wal-append.queue_time_us":230722}
I20250116 02:59:32.660584 20723 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:32.731281 23063 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:32.783458 20750 raft_consensus.cc:2388] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:32.777738 22914 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:32.783010 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 163807us
I20250116 02:59:32.783389 23063 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.788074 23063 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:32.783833 23107 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:32.784015 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1555) took 1307 ms. Trace:
I20250116 02:59:32.789649 20750 rpcz_store.cc:276] 0116 02:59:31.476761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.604132 (+1127371us) service_pool.cc:225] Handling call
0116 02:59:32.783946 (+179814us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.769438 20756 raft_consensus.cc:2405] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250116 02:59:32.774571 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.774977 22825 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:32.620051 23126 raft_consensus.cc:695] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:32.653133 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1551) took 1312 ms. Trace:
I20250116 02:59:32.777640 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.672291 22884 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:32.789268 23107 raft_consensus.cc:695] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:32.720160 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:32.762478 20572 leader_election.cc:336] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.764s (SENT)
W20250116 02:59:32.834862 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 63535us
I20250116 02:59:32.834326 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:32.862097 20678 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:32.732843 22821 raft_consensus.cc:988] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: : Instructing follower 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
I20250116 02:59:32.768895 20752 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:32.857383 20749 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:32.863924 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:32.858897 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:32.862097 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1953) took 1038 ms. Trace:
W20250116 02:59:32.772661 20680 rpcz_store.cc:269] 0116 02:59:30.954961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.384764 (+429803us) service_pool.cc:225] Handling call
0116 02:59:32.672561 (+1287797us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54001919d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:32.704764 (+ 32203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":435839360}
I20250116 02:59:32.621856 23171 raft_consensus.cc:3054] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:32.863624 20676 raft_consensus.cc:2388] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:32.720358 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:32.807197 22936 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:32.866760 20756 rpcz_store.cc:276] 0116 02:59:31.823967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.609243 (+785276us) service_pool.cc:225] Handling call
0116 02:59:32.862018 (+252775us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:32.866010 20749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 446) took 1291 ms (client timeout 1594 ms). Trace:
W20250116 02:59:32.868803 20607 consensus_meta.cc:287] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.519s user 0.011s sys 0.000s
W20250116 02:59:32.871497 20749 rpcz_store.cc:269] 0116 02:59:31.574470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.777271 (+1202801us) service_pool.cc:225] Handling call
0116 02:59:32.865528 (+ 88257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.750237 20599 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:32.875042 20750 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:32.870972 20607 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:32.867874 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:32.879076 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 179797us
I20250116 02:59:32.768455 22927 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:32.788733 23063 consensus_queue.cc:1046] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20250116 02:59:32.947407 23126 consensus_queue.cc:237] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.914181 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.950392 20749 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:32.950975 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1946) took 1130 ms. Trace:
I20250116 02:59:32.951290 20749 rpcz_store.cc:276] 0116 02:59:31.820392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.873120 (+1052728us) service_pool.cc:225] Handling call
0116 02:59:32.950907 (+ 77787us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.672293 22957 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:32.917516 22825 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:32.924070 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.908851 20674 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:32.924857 23130 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:32.925057 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 50541.1us
I20250116 02:59:32.931561 23107 consensus_queue.cc:237] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:32.934136 22519 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:32.934676 20720 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:32.938434 23196 log.cc:926] Time spent T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
I20250116 02:59:32.938807 23113 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:32.940439 22980 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:59:32.861351 20748 rpcz_store.cc:276] 0116 02:59:31.340311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.405590 (+1065279us) service_pool.cc:225] Handling call
0116 02:59:32.635211 (+229621us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.943306 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:32.873556 22959 raft_consensus.cc:1076] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Signalling peer 57fcfcd5ab01438c9a4baf8a61871aa6 to start an election
W20250116 02:59:32.958798 20681 consensus_meta.cc:287] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.602s user 0.012s sys 0.000s
I20250116 02:59:32.985267 22825 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:32.987385 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.988247 20754 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:32.988790 20754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1553) took 1647 ms (client timeout 2009 ms). Trace:
W20250116 02:59:32.989182 20754 rpcz_store.cc:269] 0116 02:59:31.340852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.986890 (+1646038us) service_pool.cc:225] Handling call
0116 02:59:32.988717 (+ 1827us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.991190 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:32.992033 20753 raft_consensus.cc:3054] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 5
I20250116 02:59:32.964989 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:32.915043 22821 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
W20250116 02:59:32.964133 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 201703us
I20250116 02:59:32.970778 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.001376 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1945) took 1181 ms. Trace:
I20250116 02:59:33.001744 20749 rpcz_store.cc:276] 0116 02:59:31.819897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.951953 (+1132056us) service_pool.cc:225] Handling call
0116 02:59:33.001304 (+ 49351us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:32.975390 20571 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:32.972399 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1950) took 1150 ms. Trace:
I20250116 02:59:32.854388 22871 consensus_queue.cc:1035] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20250116 02:59:33.009400 20678 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:32.965181 22957 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:32.959434 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1670) took 1153 ms. Trace:
W20250116 02:59:32.991261 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 137170us
W20250116 02:59:32.996592 20647 leader_election.cc:336] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.815s (SENT)
I20250116 02:59:32.990275 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:32.972083 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:59:32.982152 21299 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:33.002885 20756 rpcz_store.cc:276] 0116 02:59:31.822543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.889278 (+1066735us) service_pool.cc:225] Handling call
0116 02:59:32.972325 (+ 83047us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:33.004030 20572 leader_election.cc:336] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.782s (SENT)
I20250116 02:59:32.984218 20681 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:33.016808 20607 rpcz_store.cc:276] 0116 02:59:31.763611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.144873 (+381262us) service_pool.cc:225] Handling call
0116 02:59:32.917137 (+772264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.024291 20607 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
W20250116 02:59:33.029662 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 47612.4us
I20250116 02:59:32.993028 22927 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:33.018486 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 231741us
I20250116 02:59:33.026048 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.049109 20754 raft_consensus.cc:2388] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:33.049451 23199 log.cc:926] Time spent T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20250116 02:59:33.049808 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1954) took 1225 ms. Trace:
I20250116 02:59:33.050377 20720 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:33.004139 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.025293 23071 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:33.064554 20748 raft_consensus.cc:2388] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:33.064627 20749 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:33.065004 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:33.065676 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 61818.8us
I20250116 02:59:33.024618 22821 raft_consensus.cc:695] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:33.026353 23115 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:33.077266 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1951) took 1254 ms. Trace:
I20250116 02:59:33.082713 20749 rpcz_store.cc:276] 0116 02:59:31.823031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.002453 (+1179422us) service_pool.cc:225] Handling call
0116 02:59:33.077140 (+ 74687us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.050304 20754 rpcz_store.cc:276] 0116 02:59:31.824401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.989930 (+1165529us) service_pool.cc:225] Handling call
0116 02:59:33.049741 (+ 59811us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.082805 20603 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:32.979425 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:33.084371 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 400) took 1064 ms. Trace:
I20250116 02:59:33.088897 20603 rpcz_store.cc:276] 0116 02:59:32.019605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.907977 (+888372us) service_pool.cc:225] Handling call
0116 02:59:33.073492 (+165515us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:33.082349 (+ 8857us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.084042 (+ 1693us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.084301 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6478464}
I20250116 02:59:33.088476 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1550) took 1748 ms. Trace:
I20250116 02:59:33.091449 20756 rpcz_store.cc:276] 0116 02:59:31.340035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.019457 (+1679422us) service_pool.cc:225] Handling call
0116 02:59:33.088404 (+ 68947us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.085471 21299 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:33.057723 23113 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:33.096114 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 118258us
I20250116 02:59:33.081789 21333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:56664
I20250116 02:59:33.015928 23130 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.066121 22825 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.104095 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.079842 20675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1515) took 1190 ms. Trace:
I20250116 02:59:33.105652 21333 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:59:33.111476 20608 raft_consensus.cc:1235] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250116 02:59:33.112396 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:33.116252 21333 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:33.111481 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 52921.7us
I20250116 02:59:33.123023 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1669) took 1353 ms. Trace:
I20250116 02:59:33.123574 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 398) took 1237 ms. Trace:
I20250116 02:59:33.123703 21299 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:33.123585 20608 rpcz_store.cc:276] 0116 02:59:31.763102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.012518 (+1249416us) service_pool.cc:225] Handling call
0116 02:59:33.066573 (+ 54055us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:33.116433 (+ 49860us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.116746 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.126380 23204 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:33.119977 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:33.127271 23204 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:33.128479 20720 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:33.065239 23071 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.123996 20682 rpcz_store.cc:276] 0116 02:59:31.886433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.248604 (+362171us) service_pool.cc:225] Handling call
0116 02:59:32.249083 (+ 479us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.283634 (+ 34551us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.122763 (+839129us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b54000870d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:33.123508 (+ 745us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54969088}
I20250116 02:59:33.125280 21299 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:33.146202 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.143007 20676 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:33.151012 20756 raft_consensus.cc:2405] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:33.145881 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.153877 20748 raft_consensus.cc:2388] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:33.154558 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1952) took 1331 ms. Trace:
I20250116 02:59:33.155031 22930 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:33.156405 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:33.155220 22895 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:33.158162 22895 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:33.158499 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1668) took 1413 ms. Trace:
I20250116 02:59:33.159096 20604 rpcz_store.cc:276] 0116 02:59:31.745217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.366457 (+621240us) service_pool.cc:225] Handling call
0116 02:59:32.373088 (+ 6631us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.479908 (+106820us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.479981 (+ 73us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:32.480108 (+ 127us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:32.480383 (+ 275us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.158233 (+677850us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b540014dcd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:33.158294 (+ 61us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.158436 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":242569472}
I20250116 02:59:33.160387 20600 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:33.161832 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1956) took 1336 ms. Trace:
I20250116 02:59:33.162297 20756 rpcz_store.cc:276] 0116 02:59:31.825308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.119340 (+1294032us) service_pool.cc:225] Handling call
0116 02:59:33.161756 (+ 42416us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":875904}
I20250116 02:59:33.168134 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.088198 22927 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:33.170032 20753 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
W20250116 02:59:33.171049 20753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 443) took 2346 ms (client timeout 2197 ms). Trace:
W20250116 02:59:33.171532 20753 rpcz_store.cc:269] 0116 02:59:30.824506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.990799 (+2166293us) service_pool.cc:225] Handling call
0116 02:59:33.170963 (+180164us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:33.150831 20647 leader_election.cc:336] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.843s (SENT)
I20250116 02:59:33.175222 23130 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.177115 22936 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.177875 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 397) took 1265 ms. Trace:
I20250116 02:59:33.158514 22895 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:33.179795 23113 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.180370 22936 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:33.181715 22908 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:33.182255 22908 raft_consensus.cc:695] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:33.087527 22821 consensus_queue.cc:237] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:33.183168 21327 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:33.145004 22753 raft_consensus.cc:2798] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:33.191846 20749 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:33.189438 20677 consensus_meta.cc:287] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.833s user 0.009s sys 0.003s
I20250116 02:59:33.194044 20677 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:33.172948 20647 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:33.154872 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.185457 22980 consensus_queue.cc:1046] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250116 02:59:33.197171 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.207398 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:33.208304 20753 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:33.217114 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 50206.9us
W20250116 02:59:33.221374 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 90936.3us
I20250116 02:59:33.222355 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.216228 21333 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.202126 20756 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:33.115407 20675 rpcz_store.cc:276] 0116 02:59:31.884449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.949491 (+ 65042us) service_pool.cc:225] Handling call
0116 02:59:33.041262 (+1091771us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b54001923d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:33.074230 (+ 32968us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":605893504}
W20250116 02:59:33.204061 22791 consensus_meta.cc:287] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.851s user 0.010s sys 0.005s
I20250116 02:59:33.198308 22959 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:33.162356 20748 rpcz_store.cc:276] 0116 02:59:31.823508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.093213 (+1269705us) service_pool.cc:225] Handling call
0116 02:59:33.154461 (+ 61248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.232231 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.219419 20754 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
W20250116 02:59:33.235811 20754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1540) took 2556 ms (client timeout 3300 ms). Trace:
W20250116 02:59:33.236331 20754 rpcz_store.cc:269] 0116 02:59:30.679405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.092616 (+2413211us) service_pool.cc:225] Handling call
0116 02:59:33.235737 (+143121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.237502 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:33.238741 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 426196us
I20250116 02:59:33.238830 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:33.239573 20749 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:33.240800 21300 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35540 dropped due to backpressure: service queue is full with 50 items [suppressed 43 similar messages]
I20250116 02:59:33.241603 21300 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:33.238536 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1948) took 1417 ms. Trace:
I20250116 02:59:33.243048 20754 rpcz_store.cc:276] 0116 02:59:31.821424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.237124 (+1415700us) service_pool.cc:225] Handling call
0116 02:59:33.238477 (+ 1353us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.243853 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.244601 20754 raft_consensus.cc:2388] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:33.245308 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:33.243667 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:33.245987 20754 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:33.246398 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1943) took 1428 ms. Trace:
I20250116 02:59:33.246771 20754 rpcz_store.cc:276] 0116 02:59:31.818548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.245092 (+1426544us) service_pool.cc:225] Handling call
0116 02:59:33.246352 (+ 1260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.247414 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.234020 22843 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:33.248337 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 428) took 4663 ms. Trace:
I20250116 02:59:33.249100 22843 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.250080 20754 rpcz_store.cc:276] 0116 02:59:28.585362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.247412 (+4662050us) service_pool.cc:225] Handling call
0116 02:59:33.247791 (+ 379us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.248007 (+ 216us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.248034 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.248082 (+ 48us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:33.248120 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.248173 (+ 53us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.248272 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.248956 20756 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:33.251657 22843 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.252050 20754 raft_consensus.cc:1235] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250116 02:59:33.178391 20602 rpcz_store.cc:276] 0116 02:59:31.914850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.176831 (+1261981us) service_pool.cc:225] Handling call
0116 02:59:33.177241 (+ 410us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.177516 (+ 275us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.177546 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.177598 (+ 52us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:33.177632 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.177692 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.177811 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.252804 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 429) took 4643 ms. Trace:
I20250116 02:59:33.253149 20754 rpcz_store.cc:276] 0116 02:59:28.609870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.251350 (+4641480us) service_pool.cc:225] Handling call
0116 02:59:33.251814 (+ 464us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:33.252480 (+ 666us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.252717 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.254144 22843 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:33.254881 22843 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.255424 20720 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:33.255775 23071 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.257151 20749 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:33.257275 22843 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.259821 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1918) took 4580 ms. Trace:
I20250116 02:59:33.263322 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:33.263048 22908 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.270182 20749 rpcz_store.cc:276] 0116 02:59:28.682305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.258519 (+4576214us) service_pool.cc:225] Handling call
0116 02:59:33.259012 (+ 493us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.259456 (+ 444us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.259485 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.259534 (+ 49us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:33.259579 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.259638 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.259752 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.264937 22998 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:33.220327 22753 raft_consensus.cc:695] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:33.225090 22519 raft_consensus.cc:3049] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Stepping down as leader of term 6
W20250116 02:59:33.266239 20751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1509) took 3242 ms (client timeout 1959 ms). Trace:
I20250116 02:59:33.266467 20607 raft_consensus.cc:1270] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250116 02:59:33.268980 20603 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:33.264233 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:33.232508 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 152800us
W20250116 02:59:33.307546 20751 rpcz_store.cc:269] 0116 02:59:30.024060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.492475 (+1468415us) service_pool.cc:225] Handling call
0116 02:59:33.265977 (+1773502us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b54000cc6d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:33.266163 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1409366400}
I20250116 02:59:33.313110 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 401) took 1275 ms. Trace:
I20250116 02:59:33.313599 20606 rpcz_store.cc:276] 0116 02:59:32.037299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.290825 (+253526us) service_pool.cc:225] Handling call
0116 02:59:33.312864 (+1022039us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b54002544d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:33.313032 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1386869120}
W20250116 02:59:33.314680 20757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1537) took 2716 ms (client timeout 1940 ms). Trace:
W20250116 02:59:33.315145 20757 rpcz_store.cc:269] 0116 02:59:30.597868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:31.538441 (+940573us) service_pool.cc:225] Handling call
0116 02:59:33.280580 (+1742139us) spinlock_profiling.cc:243] Waited 962 ms on lock 0x7b54000cd0d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:33.314567 (+ 33987us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2212457344}
I20250116 02:59:33.315673 22753 consensus_queue.cc:237] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.316955 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1940) took 4570 ms. Trace:
I20250116 02:59:33.317356 20757 rpcz_store.cc:276] 0116 02:59:28.747027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.315996 (+4568969us) service_pool.cc:225] Handling call
0116 02:59:33.316404 (+ 408us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.316628 (+ 224us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.316654 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.316695 (+ 41us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:33.316747 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.316801 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.316897 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.318869 21333 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:33.243712 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:33.321991 20748 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:33.322552 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1957) took 1027 ms. Trace:
I20250116 02:59:33.315696 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 403) took 1212 ms. Trace:
I20250116 02:59:33.323400 20599 rpcz_store.cc:276] 0116 02:59:32.103111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.928208 (+825097us) service_pool.cc:225] Handling call
0116 02:59:33.314826 (+386618us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:33.315061 (+ 235us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.315326 (+ 265us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.315359 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.315407 (+ 48us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:33.315448 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.315516 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.315633 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55467392}
I20250116 02:59:33.324558 22895 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.326473 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:33.327399 20604 raft_consensus.cc:2363] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:33.328060 20720 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:33.329524 23063 consensus_queue.cc:1046] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:33.307054 22519 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:33.311426 22998 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:33.338074 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 362796us
I20250116 02:59:33.338874 21299 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:33.245543 22936 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:33.340762 20752 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
W20250116 02:59:33.348595 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1944) took 1529 ms (client timeout 1874 ms). Trace:
W20250116 02:59:33.348939 20752 rpcz_store.cc:269] 0116 02:59:31.819292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.641542 (+822250us) service_pool.cc:225] Handling call
0116 02:59:33.348539 (+706997us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.267335 22927 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:33.345371 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 202177us
I20250116 02:59:33.354211 20749 raft_consensus.cc:1235] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
W20250116 02:59:33.361358 23190 log.cc:926] Time spent T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20250116 02:59:33.362512 20647 leader_election.cc:336] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.909s (SENT)
W20250116 02:59:33.362972 23203 log.cc:926] Time spent T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20250116 02:59:33.362845 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1602) took 1614 ms. Trace:
I20250116 02:59:33.363095 20647 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:33.364025 20572 leader_election.cc:336] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.903s (SENT)
I20250116 02:59:33.367868 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 432) took 4557 ms. Trace:
I20250116 02:59:33.368927 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:33.369832 20678 raft_consensus.cc:3049] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:33.339407 22957 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.370543 20678 raft_consensus.cc:738] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:33.359102 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:33.368570 20752 rpcz_store.cc:276] 0116 02:59:28.810528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.349936 (+4539408us) service_pool.cc:225] Handling call
0116 02:59:33.367102 (+ 17166us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.367343 (+ 241us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.367368 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.367479 (+ 111us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:33.367576 (+ 97us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.367644 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.367773 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:33.377465 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 320194us
I20250116 02:59:33.306905 22959 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:33.385337 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1924) took 4642 ms. Trace:
I20250116 02:59:33.385825 20749 rpcz_store.cc:276] 0116 02:59:28.718093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.306758 (+4588665us) service_pool.cc:225] Handling call
0116 02:59:33.353993 (+ 47235us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.360255 (+ 6262us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.360518 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.381155 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 431) took 4550 ms. Trace:
I20250116 02:59:33.394198 20757 rpcz_store.cc:276] 0116 02:59:28.805582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.318265 (+4512683us) service_pool.cc:225] Handling call
0116 02:59:33.334905 (+ 16640us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.348163 (+ 13258us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.348190 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.355190 (+ 7000us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:33.355243 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.355328 (+ 85us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.355488 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.383855 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1939) took 4628 ms. Trace:
I20250116 02:59:33.395977 20751 rpcz_store.cc:276] 0116 02:59:28.746739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.313161 (+4566422us) service_pool.cc:225] Handling call
0116 02:59:33.350931 (+ 37770us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.374912 (+ 23981us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.374946 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.375011 (+ 65us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:33.375063 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.375134 (+ 71us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.375271 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.391757 20723 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:33.347914 22936 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:33.363467 20683 rpcz_store.cc:276] 0116 02:59:31.748042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.076359 (+328317us) service_pool.cc:225] Handling call
0116 02:59:32.107571 (+ 31212us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:32.153757 (+ 46186us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:32.153790 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:32.161525 (+ 7735us) log.cc:844] Serialized 26 byte log entry
0116 02:59:32.363021 (+201496us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:32.363060 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:32.363124 (+ 64us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:33.362579 (+999455us) raft_consensus.cc:1680] finished
0116 02:59:33.362633 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.362783 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16079,"raft.queue_time_us":25309,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":142,"thread_start_us":82337,"threads_started":1,"wal-append.queue_time_us":831071}
I20250116 02:59:33.368280 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:33.392740 20603 raft_consensus.cc:1235] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250116 02:59:33.406940 21333 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:33.372398 22957 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.368585 22994 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:33.322898 20748 rpcz_store.cc:276] 0116 02:59:32.295594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.243283 (+947689us) service_pool.cc:225] Handling call
0116 02:59:33.322478 (+ 79195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15744}
I20250116 02:59:33.391396 22927 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:33.374294 22868 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:33.403903 22980 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:33.413908 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:33.443164 20604 raft_consensus.cc:3054] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 6
I20250116 02:59:33.374759 22930 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:33.371750 20678 consensus_queue.cc:260] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:33.442013 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.026s sys 0.033s
I20250116 02:59:33.354735 20571 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:33.404675 22936 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:33.420630 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 434) took 4580 ms. Trace:
I20250116 02:59:33.414008 22994 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:33.423589 20751 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250116 02:59:33.445425 20753 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
W20250116 02:59:33.446792 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.027s sys 0.033s
W20250116 02:59:33.453367 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 663291us
I20250116 02:59:33.447850 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 435) took 4214 ms. Trace:
I20250116 02:59:33.449244 20752 rpcz_store.cc:276] 0116 02:59:28.839917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.393543 (+4553626us) service_pool.cc:225] Handling call
0116 02:59:33.419990 (+ 26447us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.420246 (+ 256us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.420273 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.420321 (+ 48us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:33.420367 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.420425 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.420557 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.425338 20723 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:33.454238 20757 rpcz_store.cc:276] 0116 02:59:29.233412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.403370 (+4169958us) service_pool.cc:225] Handling call
0116 02:59:33.440518 (+ 37148us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.447471 (+ 6953us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.447502 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.447550 (+ 48us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:33.447589 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.447652 (+ 63us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.447781 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:33.451784 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 97146.4us
I20250116 02:59:33.452878 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 433) took 4619 ms. Trace:
I20250116 02:59:33.455977 20749 rpcz_store.cc:276] 0116 02:59:28.833976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.386745 (+4552769us) service_pool.cc:225] Handling call
0116 02:59:33.402008 (+ 15263us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.442383 (+ 40375us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.442417 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.448382 (+ 5965us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:33.452597 (+ 4215us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.452676 (+ 79us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.452811 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.415097 20602 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:33.460664 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:33.461548 20683 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 6
I20250116 02:59:33.463917 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 437) took 4129 ms. Trace:
I20250116 02:59:33.464433 20751 rpcz_store.cc:276] 0116 02:59:29.334573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.403412 (+4068839us) service_pool.cc:225] Handling call
0116 02:59:33.423341 (+ 19929us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:33.463513 (+ 40172us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.463832 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:33.443667 23171 consensus_meta.cc:287] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.534s user 0.008s sys 0.004s
I20250116 02:59:33.468209 20723 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:33.466974 21302 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:33.468508 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1960) took 1160 ms. Trace:
I20250116 02:59:33.469120 20753 rpcz_store.cc:276] 0116 02:59:32.296598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.206984 (+910386us) service_pool.cc:225] Handling call
0116 02:59:33.456526 (+249542us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.469803 21302 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:33.461131 22994 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:33.443060 23210 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:33.405203 22959 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.495656 20753 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:33.477967 20755 raft_consensus.cc:2405] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250116 02:59:33.460005 22957 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:33.483197 20571 leader_election.cc:400] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:33.497987 20571 leader_election.cc:403] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250116 02:59:33.485214 20751 raft_consensus.cc:1235] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20250116 02:59:33.487813 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 125340us
I20250116 02:59:33.498178 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 445) took 1925 ms. Trace:
W20250116 02:59:33.488881 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 247361us
I20250116 02:59:33.492305 20757 raft_consensus.cc:1270] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250116 02:59:33.469906 22895 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:33.447467 22868 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.493306 20752 raft_consensus.cc:1235] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250116 02:59:33.492342 20749 raft_consensus.cc:1270] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250116 02:59:33.478835 22843 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:33.468968 22927 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250116 02:59:33.499926 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1521) took 3472 ms. Trace:
I20250116 02:59:33.499573 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 439) took 3264 ms. Trace:
I20250116 02:59:33.499949 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1501) took 3478 ms. Trace:
I20250116 02:59:33.522953 20749 rpcz_store.cc:276] 0116 02:59:30.021866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.471419 (+3449553us) service_pool.cc:225] Handling call
0116 02:59:33.490083 (+ 18664us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.499723 (+ 9640us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.499893 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.522331 20752 rpcz_store.cc:276] 0116 02:59:30.027084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.458038 (+3430954us) service_pool.cc:225] Handling call
0116 02:59:33.483649 (+ 25611us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:33.499649 (+ 16000us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.499859 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.499085 20753 rpcz_store.cc:276] 0116 02:59:31.572531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.478883 (+1906352us) service_pool.cc:225] Handling call
0116 02:59:33.495492 (+ 16609us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.497852 (+ 2360us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.498107 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.516913 23205 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:33.518019 23063 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:33.516748 23209 raft_consensus.cc:2798] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:33.519425 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1524) took 3491 ms. Trace:
W20250116 02:59:33.521806 20750 consensus_meta.cc:287] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.599s user 0.009s sys 0.000s
I20250116 02:59:33.522449 20757 rpcz_store.cc:276] 0116 02:59:30.235216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.471461 (+3236245us) service_pool.cc:225] Handling call
0116 02:59:33.487301 (+ 15840us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.499299 (+ 11998us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.499515 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:33.526254 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 60142us
I20250116 02:59:33.541460 20683 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:33.469405 22519 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:33.553442 20647 leader_election.cc:336] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.042s (SENT)
I20250116 02:59:33.561802 20647 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:33.562072 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 448) took 1463 ms. Trace:
I20250116 02:59:33.545003 20750 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
W20250116 02:59:33.562916 20750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 444) took 1995 ms (client timeout 2158 ms). Trace:
W20250116 02:59:33.563297 20750 rpcz_store.cc:269] 0116 02:59:31.567009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.790481 (+1223472us) service_pool.cc:225] Handling call
0116 02:59:33.562835 (+772354us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.562454 20752 rpcz_store.cc:276] 0116 02:59:32.099272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.524570 (+1425298us) service_pool.cc:225] Handling call
0116 02:59:33.531795 (+ 7225us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.546041 (+ 14246us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.546073 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.561725 (+ 15652us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:33.561773 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.561853 (+ 80us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.562006 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:33.518893 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 190980us
I20250116 02:59:33.540166 20751 rpcz_store.cc:276] 0116 02:59:30.027840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.465291 (+3437451us) service_pool.cc:225] Handling call
0116 02:59:33.475292 (+ 10001us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:33.500647 (+ 25355us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.519352 (+ 18705us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.546089 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 438) took 3726 ms. Trace:
I20250116 02:59:33.551215 21299 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:33.550753 22843 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.525657 22927 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:33.554735 23210 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:33.532683 23209 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:33.528765 23063 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:33.527971 23126 raft_consensus.cc:2743] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
W20250116 02:59:33.578279 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 53880.5us
I20250116 02:59:33.545594 22895 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:33.625216 20572 leader_election.cc:336] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.537s (SENT)
W20250116 02:59:33.627584 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 50821.4us
W20250116 02:59:33.650369 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 161767us
W20250116 02:59:33.627267 23213 log.cc:926] Time spent T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20250116 02:59:33.626370 22993 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
W20250116 02:59:33.651757 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 183679us
I20250116 02:59:33.654299 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 430) took 4977 ms. Trace:
I20250116 02:59:33.628738 22821 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:33.655694 20571 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:33.638885 23063 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:33.627074 23209 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:33.577476 22753 consensus_queue.cc:1046] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20250116 02:59:33.655972 20756 rpcz_store.cc:276] 0116 02:59:28.673260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.254809 (+4581549us) service_pool.cc:225] Handling call
0116 02:59:33.255208 (+ 399us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.255511 (+ 303us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:33.475023 (+219512us) log.cc:844] Serialized 14 byte log entry
0116 02:59:33.649751 (+174728us) log.cc:844] Serialized 14 byte log entry
0116 02:59:33.649914 (+ 163us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.649959 (+ 45us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:33.650006 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.650075 (+ 69us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.650210 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8658,"raft.queue_time_us":12547,"thread_start_us":367455,"threads_started":2}
I20250116 02:59:33.576501 22871 consensus_queue.cc:1046] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20250116 02:59:33.649662 23218 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:33.649185 22927 raft_consensus.cc:695] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
W20250116 02:59:33.687927 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 62766.1us
W20250116 02:59:33.693156 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 139734us
I20250116 02:59:33.691419 20571 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:33.693852 20647 leader_election.cc:336] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.122s (SENT)
I20250116 02:59:33.698777 20647 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:33.699105 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:33.675611 23126 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:33.688726 20754 raft_consensus.cc:2949] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:33.689447 22959 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.575968 20748 rpcz_store.cc:276] 0116 02:59:29.819539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.446239 (+3626700us) service_pool.cc:225] Handling call
0116 02:59:33.477518 (+ 31279us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.493043 (+ 15525us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.497250 (+ 4207us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.501000 (+ 3750us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:33.522793 (+ 21793us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.528699 (+ 5906us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.543156 (+ 14457us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:33.710996 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:33.701195 22994 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.695478 23218 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:33.756997 20572 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.300s (SENT)
I20250116 02:59:33.851365 20720 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:33.738540 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 51666.4us
I20250116 02:59:33.822527 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:33.863467 20681 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 02:59:33.787006 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 93173.4us
W20250116 02:59:33.755877 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 72113.5us
I20250116 02:59:33.865259 21300 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:33.806560 20571 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:33.716553 23113 raft_consensus.cc:2798] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:33.710989 22843 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.887688 23218 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:33.888379 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 81927.6us
I20250116 02:59:33.975173 20571 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:33.720970 22959 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:33.790736 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:33.784372 23210 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:33.889612 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 137535us
W20250116 02:59:33.891161 23224 log.cc:926] Time spent T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
W20250116 02:59:33.891461 23226 log.cc:926] Time spent T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20250116 02:59:33.891927 23171 consensus_queue.cc:1046] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:59:33.893569 23235 raft_consensus.cc:2798] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:33.895556 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:33.753615 22993 raft_consensus.cc:695] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:33.806883 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 404) took 1180 ms. Trace:
I20250116 02:59:33.899574 23126 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:33.902839 23208 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:33.912321 20647 leader_election.cc:336] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.289s (SENT)
I20250116 02:59:33.914952 22980 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:34.001371 20572 leader_election.cc:336] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.575s (SENT)
I20250116 02:59:33.924683 23212 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:33.924887 22998 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:33.929572 21328 raft_consensus.cc:1235] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:33.931178 20678 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:33.935467 20676 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34864
I20250116 02:59:33.937628 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.009073 21333 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:33.940829 20755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1547) took 2659 ms (client timeout 1890 ms). Trace:
I20250116 02:59:33.943123 22825 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:33.943415 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 402) took 1862 ms. Trace:
I20250116 02:59:33.945524 22791 consensus_queue.cc:1046] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:33.946584 23207 raft_consensus.cc:3049] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:33.950671 22519 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:34.025481 21299 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:33.957919 20604 consensus_meta.cc:287] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.509s user 0.011s sys 0.000s
I20250116 02:59:33.924508 23222 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:33.960542 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 2048) took 1358 ms. Trace:
I20250116 02:59:33.950706 22936 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.031455 20605 rpcz_store.cc:276] 0116 02:59:32.601739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.641909 (+ 40170us) service_pool.cc:225] Handling call
0116 02:59:33.960254 (+1318345us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b5400108e54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:33.960448 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1664476544}
I20250116 02:59:33.961336 23083 raft_consensus.cc:3049] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:33.957640 20603 raft_consensus.cc:1270] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:34.010955 20601 rpcz_store.cc:276] 0116 02:59:32.081224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.919299 (+838075us) service_pool.cc:225] Handling call
0116 02:59:32.969539 (+ 50240us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:32.991050 (+ 21511us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:32.991436 (+ 386us) log.cc:844] Serialized 14 byte log entry
0116 02:59:33.353911 (+362475us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:33.353974 (+ 63us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:33.354116 (+ 142us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.943141 (+589025us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b540014d2d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:33.943203 (+ 62us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.943348 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":809,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":57,"raft.queue_time_us":14129,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":99,"spinlock_wait_cycles":48729856,"thread_start_us":361470,"threads_started":1,"wal-append.queue_time_us":484112}
I20250116 02:59:33.962133 23063 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:33.963346 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 152665us
I20250116 02:59:33.963796 20649 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:33.996688 23210 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:33.761659 22927 consensus_queue.cc:237] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:33.755039 22957 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:33.789971 23227 consensus_queue.cc:1035] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.184s
I20250116 02:59:33.754772 22868 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:33.809165 23071 consensus_queue.cc:1046] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.172s
W20250116 02:59:34.000062 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 109266us
I20250116 02:59:33.770936 23205 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.212s
I20250116 02:59:34.004428 21302 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:34.006135 20647 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:33.997817 20679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:33.891882 23228 consensus_queue.cc:1046] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20250116 02:59:34.010552 20646 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:34.060343 23246 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:34.065346 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 153060us
I20250116 02:59:34.066257 23228 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:34.065831 22791 raft_consensus.cc:2743] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:34.064496 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 60176.8us
I20250116 02:59:34.068017 23209 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.065105 20679 raft_consensus.cc:604] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:34.069597 20679 raft_consensus.cc:612] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:34.071027 22993 consensus_queue.cc:237] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.045944 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:34.072360 20600 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
W20250116 02:59:34.064266 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 100714us
I20250116 02:59:33.973560 20504 catalog_manager.cc:5526] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 3, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } }
I20250116 02:59:34.028752 20604 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:34.028945 20572 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:34.010151 20681 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250116 02:59:34.010089 20755 rpcz_store.cc:269] 0116 02:59:31.281730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.490984 (+1209254us) service_pool.cc:225] Handling call
0116 02:59:33.940586 (+1449602us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b5400396054. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:33.940743 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2180404096}
I20250116 02:59:33.974591 22930 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20250116 02:59:34.046447 20678 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:34.011112 23235 raft_consensus.cc:695] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:34.027379 23207 raft_consensus.cc:738] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:34.084983 23063 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:34.085983 22980 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:34.053471 21327 raft_consensus.cc:1235] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:34.039027 23204 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:34.039481 23218 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:34.037966 23083 raft_consensus.cc:738] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:33.988829 20599 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:34.039551 23212 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.054513 22959 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250116 02:59:34.085309 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 74861us
I20250116 02:59:34.065200 23071 raft_consensus.cc:3049] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Stepping down as leader of term 5
W20250116 02:59:34.053812 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 46118.1us
I20250116 02:59:34.010707 22825 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.057852 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.078864 20504 catalog_manager.cc:5526] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:34.080473 20676 raft_consensus.cc:604] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Received request to transfer leadership
I20250116 02:59:34.085603 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1928) took 5359 ms. Trace:
I20250116 02:59:34.083596 23242 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:34.085227 23207 consensus_queue.cc:260] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.101590 20676 raft_consensus.cc:612] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:34.058223 22957 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:34.083988 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 82649.2us
I20250116 02:59:34.089879 20649 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:34.109354 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 447) took 2534 ms. Trace:
I20250116 02:59:34.102066 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.101948 20754 rpcz_store.cc:276] 0116 02:59:28.726539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.306334 (+4579795us) service_pool.cc:225] Handling call
0116 02:59:33.347247 (+ 40913us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:33.350574 (+ 3327us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.350601 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:33.451106 (+100505us) log.cc:844] Serialized 572 byte log entry
0116 02:59:33.678774 (+227668us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:33.826961 (+148187us) log.cc:844] Serialized 14 byte log entry
0116 02:59:33.867650 (+ 40689us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.867738 (+ 88us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:34.085386 (+217648us) raft_consensus.cc:1680] finished
0116 02:59:34.085404 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:34.085528 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":95598,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":227996,"threads_started":2,"wal-append.queue_time_us":236528}
I20250116 02:59:34.099880 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.056679 23113 raft_consensus.cc:695] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:34.111701 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.065901 23246 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.110111 20649 leader_election.cc:400] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Vote denied by peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195) with higher term. Message: Invalid argument: T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:34.122977 20649 leader_election.cc:403] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20250116 02:59:34.118522 20572 leader_election.cc:336] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.600s (SENT)
I20250116 02:59:34.123606 20572 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:34.120774 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:34.124572 20682 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:34.127308 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.129176 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:34.109899 20749 rpcz_store.cc:276] 0116 02:59:31.575395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.523802 (+1948407us) service_pool.cc:225] Handling call
0116 02:59:33.548814 (+ 25012us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:33.549135 (+ 321us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:33.708183 (+159048us) log.cc:844] Serialized 14 byte log entry
0116 02:59:34.108957 (+400774us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:34.109019 (+ 62us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:34.109074 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:34.109148 (+ 74us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:34.109291 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40283,"prepare.queue_time_us":150529,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"thread_start_us":456137,"threads_started":2}
I20250116 02:59:34.086587 22980 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:34.100481 22825 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:34.085686 22791 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:34.145991 20571 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:34.140357 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 67523.3us
I20250116 02:59:34.101958 20504 catalog_manager.cc:5526] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 3, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 3 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:59:34.141417 20678 raft_consensus.cc:2388] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:34.094380 22994 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:34.095017 23071 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:34.128235 20679 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:34.142870 20649 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:34.090516 23212 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:34.095774 23249 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:34.115360 22957 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.087683 23218 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:34.129993 20681 raft_consensus.cc:604] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:34.118041 22959 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250116 02:59:34.173913 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 64706.4us
I20250116 02:59:34.090795 22871 consensus_queue.cc:1035] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20250116 02:59:34.142107 22895 raft_consensus.cc:3049] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:34.144613 22980 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:34.152869 23230 log.cc:926] Time spent T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.135s user 0.001s sys 0.001s
I20250116 02:59:34.147912 22821 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:34.181972 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:34.176432 20504 catalog_manager.cc:5526] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 4, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:34.147526 22843 raft_consensus.cc:2743] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:34.194523 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:34.145195 23209 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:34.147694 23221 raft_consensus.cc:3049] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:34.147200 23126 raft_consensus.cc:695] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:34.190835 23233 log.cc:926] Time spent T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20250116 02:59:34.183351 22895 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:34.207713 23126 consensus_queue.cc:237] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.209779 20605 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:34.202927 23221 raft_consensus.cc:738] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:34.191329 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:34.214586 20599 raft_consensus.cc:2388] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:34.094226 23235 consensus_queue.cc:237] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.215103 22871 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250116 02:59:34.215931 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2050) took 1556 ms. Trace:
I20250116 02:59:34.216341 20608 rpcz_store.cc:276] 0116 02:59:32.659441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.125072 (+465631us) service_pool.cc:225] Handling call
0116 02:59:33.149965 (+ 24893us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:33.152455 (+ 2490us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:33.485357 (+332902us) log.cc:844] Serialized 14 byte log entry
0116 02:59:33.689342 (+203985us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:33.783660 (+ 94318us) log.cc:844] Serialized 26 byte log entry
0116 02:59:33.783884 (+ 224us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:33.784211 (+ 327us) log.cc:844] Serialized 14 byte log entry
0116 02:59:33.784417 (+ 206us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.784510 (+ 93us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:34.215684 (+431174us) raft_consensus.cc:1680] finished
0116 02:59:34.215712 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:34.215855 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7894,"prepare.queue_time_us":12496,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":75,"raft.queue_time_us":32325,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":6354,"thread_start_us":477014,"threads_started":2,"wal-append.queue_time_us":296954}
I20250116 02:59:34.214672 23221 consensus_queue.cc:260] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.196084 23238 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20250116 02:59:34.219973 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.220752 20679 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:34.174508 22959 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:34.192482 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.222893 22959 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.192596 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:34.122458 23242 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:34.087495 23204 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.194293 20600 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:59:34.173667 23218 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.200567 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:34.227563 20603 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.201589 20723 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:34.228554 23063 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:34.228941 23063 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:34.229410 23063 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.152035 22791 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.205044 20682 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:34.088307 23083 consensus_queue.cc:260] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.166162 23212 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:34.181450 22936 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:34.206350 20504 catalog_manager.cc:5526] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:34.384109 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:34.390969 20608 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:34.254781 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 67895.5us
W20250116 02:59:34.254825 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 54652.8us
I20250116 02:59:34.394065 20681 raft_consensus.cc:2388] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:33.999948 20674 rpcz_store.cc:276] 0116 02:59:32.622388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:32.997712 (+375324us) service_pool.cc:225] Handling call
0116 02:59:33.035048 (+ 37336us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:33.035575 (+ 527us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:33.035613 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:33.036212 (+ 599us) log.cc:844] Serialized 26 byte log entry
0116 02:59:33.353896 (+317684us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:33.370501 (+ 16605us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:33.370635 (+ 134us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:33.781451 (+410816us) raft_consensus.cc:1680] finished
0116 02:59:33.782712 (+ 1261us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:33.802765 (+ 20053us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9932,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":11310,"thread_start_us":304832,"threads_started":1,"wal-append.queue_time_us":486506}
I20250116 02:59:34.179934 23210 raft_consensus.cc:695] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:34.264706 20604 raft_consensus.cc:1235] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:34.417200 20504 catalog_manager.cc:5526] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:59:34.232575 20675 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:34.439736 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:34.441794 20571 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:34.344332 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:34.440920 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:34.232080 22791 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:34.198556 23246 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:34.439873 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 228763us
W20250116 02:59:34.453758 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 310977us
I20250116 02:59:34.454813 20649 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:34.455513 20572 leader_election.cc:336] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.985s (SENT)
I20250116 02:59:34.459620 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 827) took 1206 ms. Trace:
I20250116 02:59:34.460120 21327 rpcz_store.cc:276] 0116 02:59:33.253379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.253544 (+ 165us) service_pool.cc:225] Handling call
0116 02:59:33.253917 (+ 373us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:34.086925 (+833008us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:34.459352 (+372427us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b540076a7d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:34.459553 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1838120320}
I20250116 02:59:34.461259 22825 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:34.461705 22825 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:34.462054 22825 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.445240 20605 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:34.456104 20572 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:34.446113 20504 catalog_manager.cc:5526] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 4, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:34.227885 22980 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.232844 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.375670 22959 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.450976 23208 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:34.474283 23204 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:34.468961 21299 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:34.474471 20677 raft_consensus.cc:2388] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:34.475353 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:34.480307 20571 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:34.453009 20607 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:34.480350 21333 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.442338 23221 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.465844 20504 catalog_manager.cc:5526] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:34.294883 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:34.467666 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:34.487154 20601 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:34.483294 20504 catalog_manager.cc:5526] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 4, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:34.468632 22871 consensus_queue.cc:1046] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:34.486361 22843 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:34.492987 22884 consensus_queue.cc:1035] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 02:59:34.493598 22791 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:34.470453 21328 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:34.283654 23238 consensus_queue.cc:1046] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.201s
I20250116 02:59:34.230502 22821 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195) with higher term. Message: Invalid argument: T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:34.501569 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:34.502700 20681 raft_consensus.cc:604] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:34.295298 23241 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:34.494050 22791 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:34.504904 22791 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.506896 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:34.507602 20749 raft_consensus.cc:3054] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:34.507930 22994 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:34.508468 22994 raft_consensus.cc:695] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:34.509452 22994 consensus_queue.cc:237] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.476773 23210 consensus_queue.cc:237] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.485653 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:34.463081 23171 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20250116 02:59:34.470029 23083 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.524458 20646 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:34.525466 20754 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:34.488281 20504 catalog_manager.cc:5526] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 3, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 3 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 02:59:34.531417 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 46596.6us
W20250116 02:59:34.541456 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 85972us
I20250116 02:59:34.488207 20605 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:34.543754 20603 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:34.583729 22871 raft_consensus.cc:3049] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
W20250116 02:59:34.582420 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 58294.7us
I20250116 02:59:34.472846 22959 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:34.585057 20647 leader_election.cc:336] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.176s (SENT)
I20250116 02:59:34.496948 23236 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:34.620157 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 95776.5us
I20250116 02:59:34.470106 22980 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:34.580809 20749 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:34.622354 23256 raft_consensus.cc:988] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:34.629307 20606 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:34.632506 22821 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:34.544600 23246 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:34.641446 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:34.642738 22868 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:34.643373 22868 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.643834 22957 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.644671 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.645618 22868 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:34.646023 22957 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:34.646484 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.646981 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.647830 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.648275 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.647924 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.649250 23228 raft_consensus.cc:3049] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:34.649876 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.471784 23063 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.619930 22871 raft_consensus.cc:738] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:34.652716 23063 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:34.623628 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:34.653363 22871 consensus_queue.cc:260] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.658160 20682 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250116 02:59:34.658839 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:34.659801 22980 raft_consensus.cc:1076] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:34.658025 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.660257 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:34.654683 22959 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:34.660194 20679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a569f56f9d2143869b29c78321047d00"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:34.657495 22821 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:34.661803 20679 raft_consensus.cc:604] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:34.656847 23246 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:34.548357 23218 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:34.659847 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 75233.1us
I20250116 02:59:34.661125 20752 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:34.665448 21300 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:34.668938 20749 raft_consensus.cc:1270] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:34.669140 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:34.585139 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:34.621757 20504 catalog_manager.cc:5526] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:59:34.649752 23228 raft_consensus.cc:738] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:34.680174 21333 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:34.207713 23113 consensus_queue.cc:237] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.680233 20749 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:34.681746 20504 catalog_manager.cc:5526] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:59:34.688215 20504 catalog_manager.cc:5526] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 3, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 3 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 02:59:34.689648 20504 catalog_manager.cc:5526] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 3, leader changed from 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 3 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250116 02:59:34.683583 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 62515.8us
I20250116 02:59:34.684732 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.649812 21331 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:34.691236 20504 catalog_manager.cc:5526] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:34.662544 23246 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.634158 22884 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:34.694691 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.696944 21334 raft_consensus.cc:2363] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
W20250116 02:59:34.697451 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 65717.8us
I20250116 02:59:34.699808 20646 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:34.547040 23242 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:34.701845 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.647392 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.711966 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:34.672976 22821 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:34.724653 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:34.709497 20723 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:34.653365 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:34.678457 23245 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.045s
I20250116 02:59:34.723855 20682 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:34.545492 23241 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.694118 22908 raft_consensus.cc:2798] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
W20250116 02:59:34.746858 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 44073.1us
I20250116 02:59:34.748916 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:34.697962 20504 catalog_manager.cc:5526] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 3, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 3 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:34.747196 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:34.743551 20749 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:34.632874 22895 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.698485 20602 raft_consensus.cc:1235] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250116 02:59:34.750792 20504 catalog_manager.cc:5526] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 02:59:34.752465 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 122876us
I20250116 02:59:34.771281 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.774344 21302 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:34.661461 22959 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.777819 20679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:34.788133 20679 raft_consensus.cc:604] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:59:34.785652 20504 catalog_manager.cc:5526] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 3 to 4, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:34.732523 23242 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:34.708783 23255 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:34.790338 22908 raft_consensus.cc:695] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:34.795660 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.791034 23228 consensus_queue.cc:260] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:34.797571 23254 log.cc:926] Time spent T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.175s user 0.000s sys 0.000s
I20250116 02:59:34.730150 22957 consensus_queue.cc:1046] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20250116 02:59:34.769583 23256 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20250116 02:59:34.799805 20599 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:34.834022 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.835045 20681 raft_consensus.cc:2388] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:34.779861 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:34.725301 23126 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:34.807433 23218 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:34.686481 23236 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.812072 23241 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:34.709080 22884 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:34.818570 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 116814us
I20250116 02:59:34.809852 22825 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.807649 23205 consensus_queue.cc:1046] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:34.839946 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250116 02:59:34.814124 22895 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.807711 22930 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:34.849679 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:34.818270 22998 raft_consensus.cc:3049] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:34.822543 21330 raft_consensus.cc:2388] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:34.843765 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 152136us
I20250116 02:59:34.886734 21332 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:34.821866 22980 consensus_queue.cc:1035] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20250116 02:59:34.858009 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.821512 22993 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250116 02:59:34.893940 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:34.894953 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 120678us
I20250116 02:59:34.851888 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.874967 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:34.850649 23258 log.cc:926] Time spent T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20250116 02:59:34.876786 21300 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:34.833343 23242 raft_consensus.cc:695] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:34.874708 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:59:34.810067 22791 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.917045 22791 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:34.917812 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.844553 23126 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:34.920720 23126 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:34.849723 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 69982us
I20250116 02:59:34.923772 20683 raft_consensus.cc:2388] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:34.899418 20572 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.900s (SENT)
I20250116 02:59:34.914271 22871 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:34.924759 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.881623 22930 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:34.906518 20681 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:34.913962 22825 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:34.886432 22998 raft_consensus.cc:738] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:34.914541 23228 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:34.915397 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:59:34.907918 20647 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:34.954514 20608 consensus_meta.cc:287] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.536s user 0.005s sys 0.009s
I20250116 02:59:34.955924 20608 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
W20250116 02:59:34.926024 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 75167.7us
I20250116 02:59:34.931318 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:34.927822 22791 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:34.935659 23251 consensus_queue.cc:1035] T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20250116 02:59:34.936923 23242 consensus_queue.cc:237] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:34.939230 23236 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:34.941185 22884 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:34.904302 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:34.952522 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 75819.4us
I20250116 02:59:34.964875 22843 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:34.965361 22791 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:34.966014 23259 log.cc:926] Time spent T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20250116 02:59:34.966887 23262 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:34.967576 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 59761.9us
I20250116 02:59:34.968330 22998 consensus_queue.cc:260] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:34.969411 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 76363.8us
I20250116 02:59:34.943850 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:34.975813 22959 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:34.974280 20602 raft_consensus.cc:1270] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250116 02:59:34.979672 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:34.925308 23241 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:34.930064 22930 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:34.995121 21300 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:34.942696 22908 consensus_queue.cc:237] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:35.024899 23171 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:35.025858 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:35.024014 20647 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:35.026208 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:35.026763 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:35.027040 21335 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:35.033313 20680 raft_consensus.cc:2405] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250116 02:59:35.035868 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:34.993178 23255 raft_consensus.cc:695] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:35.053750 20571 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:35.062521 20680 raft_consensus.cc:1270] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250116 02:59:35.068629 23267 log.cc:926] Time spent T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20250116 02:59:35.047576 22821 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:35.048691 22998 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:35.079324 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:35.032343 20599 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:35.079650 23271 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250116 02:59:35.089259 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 65332.6us
I20250116 02:59:35.097275 20681 raft_consensus.cc:1270] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:59:35.097554 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:35.098215 20678 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:35.091682 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:35.079253 20751 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250116 02:59:35.104364 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.106290 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:35.107040 20677 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:35.055337 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:35.073709 22821 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:35.111497 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34864 (request call id 1172) took 1695 ms. Trace:
I20250116 02:59:35.118510 20676 rpcz_store.cc:276] 0116 02:59:33.407883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.935228 (+527345us) service_pool.cc:225] Handling call
0116 02:59:35.103463 (+1168235us) spinlock_profiling.cc:243] Waited 71.8 ms on lock 0x7b54002cc754. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d09bc3 00007fc32bc7082a 00007fc325349d14 00007fc325349ca2 00007fc325349bd1 00007fc325349b4c 00007fc325348e02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:35.103547 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":165127040}
I20250116 02:59:35.115518 20756 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:35.093039 23210 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:35.130498 23274 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250116 02:59:35.132690 20572 leader_election.cc:336] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.940s (SENT)
I20250116 02:59:35.060328 23212 consensus_queue.cc:1046] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
W20250116 02:59:35.134160 20572 leader_election.cc:336] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.034s (SENT)
I20250116 02:59:35.134651 20572 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:35.135535 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:35.137229 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:35.138283 20720 connection.cc:232] RPC call timeout handler was delayed by -1.0977897200000371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
I20250116 02:59:35.056421 22825 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20250116 02:59:35.136775 20608 raft_consensus.cc:1235] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:35.096076 22957 consensus_queue.cc:1046] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20250116 02:59:35.124369 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:35.057821 22993 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20250116 02:59:35.141916 20602 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:35.136864 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:35.146102 20605 raft_consensus.cc:2388] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:59:35.125104 20554 auto_rebalancer.cc:663] Could not move replica: Illegal state: not currently leader
I20250116 02:59:35.108969 22959 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
W20250116 02:59:35.126500 23083 consensus_meta.cc:287] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.541s user 0.008s sys 0.002s
I20250116 02:59:35.055112 23171 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:35.146845 20554 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: not currently leader
I20250116 02:59:35.125164 23255 consensus_queue.cc:237] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:35.039876 23241 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:35.142683 20608 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43854
I20250116 02:59:35.170408 20608 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:59:35.122793 20676 raft_consensus.cc:1270] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250116 02:59:35.154837 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:35.167172 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:35.141485 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 244290us
I20250116 02:59:35.166833 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.175118 20676 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250116 02:59:35.177245 20750 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:35.176996 20649 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:35.173655 20608 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:35.173583 20756 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:35.205960 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:35.212446 20680 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:35.214150 20676 raft_consensus.cc:1235] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
W20250116 02:59:35.214978 20572 leader_election.cc:336] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.203s (SENT)
I20250116 02:59:35.215118 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:35.215498 20572 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:35.173348 20754 raft_consensus.cc:2388] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:35.250588 20677 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
W20250116 02:59:35.158260 22927 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.033s sys 0.023s
I20250116 02:59:35.206900 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:35.303310 20757 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:35.306849 20649 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:35.308238 20756 raft_consensus.cc:2363] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:35.308625 20681 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:35.317724 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:35.319000 20677 raft_consensus.cc:2388] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:35.169430 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:35.155855 23210 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:35.155361 21330 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:35.325722 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.335559 20756 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:35.331207 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.335815 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:35.173388 22957 raft_consensus.cc:3049] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:35.335815 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.345829 20680 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:35.156177 23246 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:35.321425 22927 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s user 0.033s sys 0.023s
I20250116 02:59:35.353061 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.358317 20756 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:35.346503 20750 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:35.359134 23241 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:35.356848 22825 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:35.347774 22957 raft_consensus.cc:738] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:35.361342 22957 consensus_queue.cc:260] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:35.362828 22957 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:35.349146 20608 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:35.356801 20754 raft_consensus.cc:1270] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250116 02:59:35.369228 20608 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:35.369786 22930 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:35.329749 23275 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:35.373821 23275 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:35.374230 23275 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:35.357036 21299 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:35.413558 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 60868.2us
I20250116 02:59:35.404091 23266 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:35.206444 22993 consensus_queue.cc:1046] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.145s
I20250116 02:59:35.430087 20601 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:35.428210 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:35.430596 20755 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:35.433866 20750 raft_consensus.cc:1270] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250116 02:59:35.443754 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.444953 20756 raft_consensus.cc:2388] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:35.445266 22927 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:35.429689 23250 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:35.446280 23250 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:35.446799 23126 consensus_queue.cc:1035] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 02:59:35.405119 23241 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:35.448644 23250 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:35.448403 23241 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:35.430976 22930 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:35.451599 23244 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:35.452270 23244 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:35.452675 23271 consensus_queue.cc:1046] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250116 02:59:35.445967 22927 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:35.457343 22927 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:35.458495 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 101547us
W20250116 02:59:35.459061 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 285491us
W20250116 02:59:35.463038 20647 leader_election.cc:336] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.557s (SENT)
I20250116 02:59:35.355587 23212 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:35.464401 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:35.438726 20750 pending_rounds.cc:77] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 1
I20250116 02:59:35.468914 20754 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:35.470064 22993 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:35.470584 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:35.471606 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 164833us
I20250116 02:59:35.471838 20608 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:35.465461 20750 pending_rounds.cc:99] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250116 02:59:35.475468 20750 raft_consensus.cc:2883] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:35.478083 20754 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250116 02:59:35.479420 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:35.435163 23262 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:35.470742 22843 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:35.487078 22843 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:35.489708 22843 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:35.491118 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:35.491884 20608 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 5
I20250116 02:59:35.492565 23245 raft_consensus.cc:3049] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:35.404982 23269 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.125s
I20250116 02:59:35.502131 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.492985 23245 raft_consensus.cc:738] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:35.454510 23244 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:35.475006 23241 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:35.506470 20754 raft_consensus.cc:2388] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:35.515264 20608 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:35.534031 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:35.504313 22895 consensus_meta.cc:287] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.622s user 0.009s sys 0.005s
I20250116 02:59:35.536952 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.543887 20754 raft_consensus.cc:2363] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:35.546640 20608 raft_consensus.cc:2405] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250116 02:59:35.474172 23212 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.308936 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.310384 20754 raft_consensus.cc:2388] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:36.254271 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 456) took 1695 ms. Trace:
W20250116 02:59:36.311362 20572 leader_election.cc:336] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.890s (SENT)
I20250116 02:59:35.680593 20757 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:35.734125 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:35.627693 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.659267 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.311553 20755 rpcz_store.cc:276] 0116 02:59:34.559364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.584350 (+ 24986us) service_pool.cc:225] Handling call
0116 02:59:34.584671 (+ 321us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:35.437695 (+853024us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.253843 (+816148us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b540012dc54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:36.254155 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1944801152}
I20250116 02:59:35.813472 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:35.546943 22930 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:35.639473 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:35.875322 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:35.932574 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1692) took 1111 ms. Trace:
I20250116 02:59:36.346017 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1560) took 1170 ms. Trace:
W20250116 02:59:35.820000 20599 consensus_meta.cc:287] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.711s user 0.011s sys 0.000s
I20250116 02:59:36.364203 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.401185 20749 raft_consensus.cc:2388] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:36.361205 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 49875.9us
W20250116 02:59:36.240638 23280 log.cc:926] Time spent T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.881s user 0.002s sys 0.005s
I20250116 02:59:36.366582 20604 raft_consensus.cc:2388] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:35.897855 23278 log.cc:926] Time spent T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.544s user 0.008s sys 0.000s
W20250116 02:59:35.626092 22998 consensus_meta.cc:287] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.549s user 0.008s sys 0.003s
I20250116 02:59:36.071465 23207 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:35.563647 22843 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.364261 23063 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.170285 23241 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.476900 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 2073) took 1652 ms. Trace:
W20250116 02:59:36.370514 20647 leader_election.cc:336] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.717s (SENT)
I20250116 02:59:36.477559 20604 rpcz_store.cc:276] 0116 02:59:34.824087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.910525 (+ 86438us) service_pool.cc:225] Handling call
0116 02:59:35.813271 (+902746us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:36.476805 (+663534us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31102720}
W20250116 02:59:35.628545 23284 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250116 02:59:36.406270 20601 rpcz_store.cc:276] 0116 02:59:34.820752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.820989 (+ 237us) service_pool.cc:225] Handling call
0116 02:59:35.932291 (+1111302us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b54001f0ed4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:35.932497 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1250330880}
W20250116 02:59:35.565598 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 124135us
W20250116 02:59:36.410038 20572 leader_election.cc:336] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.009s (SENT)
I20250116 02:59:35.909865 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1683) took 1716 ms. Trace:
I20250116 02:59:36.093504 23256 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:36.481734 20606 rpcz_store.cc:276] 0116 02:59:34.193680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.194767 (+ 1087us) service_pool.cc:225] Handling call
0116 02:59:34.195122 (+ 355us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:34.640403 (+445281us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:35.909474 (+1269071us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54002e68d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:35.909769 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":993814912}
I20250116 02:59:36.406024 20757 rpcz_store.cc:276] 0116 02:59:35.175549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.206653 (+ 31104us) service_pool.cc:225] Handling call
0116 02:59:36.345958 (+1139305us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:35.600006 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.309725 20646 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:36.482388 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.358569 20605 raft_consensus.cc:2405] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250116 02:59:35.676893 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:36.128350 20675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1618) took 2027 ms. Trace:
W20250116 02:59:35.933862 23279 log.cc:926] Time spent T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.570s user 0.000s sys 0.007s
I20250116 02:59:35.817865 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:35.908434 21330 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:36.344390 23250 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:36.492973 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 426) took 1031 ms. Trace:
I20250116 02:59:36.492666 23250 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.493417 20604 rpcz_store.cc:276] 0116 02:59:35.461466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.479419 (+1017953us) service_pool.cc:225] Handling call
0116 02:59:36.492429 (+ 13010us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:36.492600 (+ 171us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:36.492628 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:36.492693 (+ 65us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:36.492733 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.492779 (+ 46us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:36.492923 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:36.462879 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 54094.3us
I20250116 02:59:36.494936 20604 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:36.098816 22753 raft_consensus.cc:3049] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
W20250116 02:59:35.836371 22871 consensus_meta.cc:287] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.895s user 0.010s sys 0.001s
I20250116 02:59:36.491113 22821 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250116 02:59:36.491420 23228 consensus_queue.cc:1046] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
W20250116 02:59:36.426920 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 55124us
W20250116 02:59:35.851688 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 378875us
I20250116 02:59:36.498214 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34864 (request call id 1173) took 1321 ms. Trace:
I20250116 02:59:36.498601 20683 rpcz_store.cc:276] 0116 02:59:35.176668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.176780 (+ 112us) service_pool.cc:225] Handling call
0116 02:59:36.497774 (+1320994us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001c4dd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c94d50 00007fc326d17f2a 00007fc32bc71c7e 00007fc32534e817 00007fc32534e7a2 00007fc32534e6d1 00007fc32534e64c 00007fc32534d902 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:36.498118 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2746868992}
I20250116 02:59:35.906004 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.424990 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.500113 20749 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
W20250116 02:59:36.346002 23255 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.008s sys 0.013s
I20250116 02:59:36.501101 20601 raft_consensus.cc:1270] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:36.499661 22753 raft_consensus.cc:738] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:36.501981 20751 raft_consensus.cc:2363] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
W20250116 02:59:36.502391 20751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1557) took 1362 ms (client timeout 1537 ms). Trace:
W20250116 02:59:36.502714 20751 rpcz_store.cc:269] 0116 02:59:35.139518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.140398 (+ 880us) service_pool.cc:225] Handling call
0116 02:59:35.905745 (+765347us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b58001f0598. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:36.502339 (+596594us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7146368}
I20250116 02:59:36.503216 22753 consensus_queue.cc:260] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.504624 22753 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:36.505328 23241 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:35.548314 22993 raft_consensus.cc:695] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:35.720346 23246 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.407069 20600 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
W20250116 02:59:36.096668 23287 log.cc:926] Time spent T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.533s user 0.004s sys 0.008s
I20250116 02:59:36.025270 23273 raft_consensus.cc:988] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
W20250116 02:59:35.780767 23292 log.cc:926] Time spent T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.154s user 0.003s sys 0.000s
I20250116 02:59:36.128065 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:36.285145 23248 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:35.897099 23113 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.344542 23257 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:35.823132 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:35.928531 23266 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:36.362483 22791 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:35.832414 23242 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.133313 23171 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:36.370803 23244 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.006s sys 0.005s
I20250116 02:59:36.472522 20682 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:59:36.495628 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 430) took 1033 ms. Trace:
I20250116 02:59:36.589246 20604 rpcz_store.cc:276] 0116 02:59:35.462422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.494314 (+1031892us) service_pool.cc:225] Handling call
0116 02:59:36.494597 (+ 283us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:36.495390 (+ 793us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.495579 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:36.484242 20608 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:36.491703 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.411900 23209 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:35.929193 22936 raft_consensus.cc:3049] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:36.550024 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1696) took 1069 ms. Trace:
I20250116 02:59:36.485447 21300 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:36.489735 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1695) took 1009 ms. Trace:
I20250116 02:59:36.411548 20599 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:36.682770 20600 rpcz_store.cc:276] 0116 02:59:35.480315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.813017 (+332702us) service_pool.cc:225] Handling call
0116 02:59:36.549947 (+736930us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:36.549588 23256 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.494832 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:36.501315 23255 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s user 0.008s sys 0.020s
I20250116 02:59:36.483855 20753 raft_consensus.cc:2363] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6.
I20250116 02:59:36.557986 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 429) took 1095 ms. Trace:
I20250116 02:59:36.697911 20606 rpcz_store.cc:276] 0116 02:59:35.462187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.486247 (+1024060us) service_pool.cc:225] Handling call
0116 02:59:36.553653 (+ 67406us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:36.557323 (+ 3670us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:36.557357 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:36.557480 (+ 123us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:36.557528 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.557656 (+ 128us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:36.557853 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:36.559012 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:36.699666 20755 raft_consensus.cc:2405] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250116 02:59:36.501897 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 428) took 1040 ms. Trace:
I20250116 02:59:36.700402 20601 rpcz_store.cc:276] 0116 02:59:35.461954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.483869 (+1021915us) service_pool.cc:225] Handling call
0116 02:59:36.500685 (+ 16816us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:36.501607 (+ 922us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.501832 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:35.635545 23275 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.251570 23071 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.482863 23302 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:36.555393 23083 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.025s sys 0.039s
W20250116 02:59:36.667937 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 258468us
I20250116 02:59:36.481292 22843 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:35.607530 23245 consensus_queue.cc:260] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.571432 22998 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:36.466528 22930 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.563496 23294 raft_consensus.cc:1076] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
W20250116 02:59:36.701876 20599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 2072) took 1878 ms (client timeout 2056 ms). Trace:
W20250116 02:59:36.723055 20599 rpcz_store.cc:269] 0116 02:59:34.823857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.887981 (+ 64124us) service_pool.cc:225] Handling call
0116 02:59:36.695413 (+1807432us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:36.723635 23304 log.cc:926] Time spent T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250116 02:59:36.550913 22868 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.724458 22868 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.494973 23250 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:36.560418 23290 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.008s sys 0.008s
W20250116 02:59:36.727200 23290 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.825s user 0.012s sys 0.008s
I20250116 02:59:36.666836 20604 raft_consensus.cc:1235] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:36.728443 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1694) took 1249 ms. Trace:
I20250116 02:59:36.728788 20604 rpcz_store.cc:276] 0116 02:59:35.479532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.590300 (+1110768us) service_pool.cc:225] Handling call
0116 02:59:36.666594 (+ 76294us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:36.728143 (+ 61549us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.728379 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:36.730162 20604 raft_consensus.cc:1235] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 1. Current term is 2. Ops: []
I20250116 02:59:36.656106 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.730906 20751 raft_consensus.cc:2371] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4. Re-sending same reply.
I20250116 02:59:36.731932 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:36.732576 20751 raft_consensus.cc:2371] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4. Re-sending same reply.
I20250116 02:59:36.666043 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:36.734360 20757 raft_consensus.cc:2363] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:36.735320 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.736325 20757 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:36.586764 23266 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:36.635913 23244 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.007s user 0.009s sys 0.009s
I20250116 02:59:36.754554 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:59:36.481736 23063 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.696216 22993 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.589466 21331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56674 (request call id 474) took 1722 ms. Trace:
W20250116 02:59:36.683324 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 198087us
W20250116 02:59:36.483698 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 174075us
W20250116 02:59:36.489209 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 143379us
I20250116 02:59:36.683908 20605 rpcz_store.cc:276] 0116 02:59:35.479871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.662982 (+183111us) service_pool.cc:225] Handling call
0116 02:59:36.489663 (+826681us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:36.705833 20750 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:36.680810 21327 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:36.572211 23248 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.717849 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1688) took 2085 ms. Trace:
I20250116 02:59:36.680702 23273 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:36.720182 20572 leader_election.cc:336] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.018s (SENT)
I20250116 02:59:36.737617 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.737973 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34870 (request call id 1529) took 1703 ms. Trace:
I20250116 02:59:36.779095 20603 rpcz_store.cc:276] 0116 02:59:34.632089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.656010 (+ 23921us) service_pool.cc:225] Handling call
0116 02:59:34.656357 (+ 347us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:34.677330 (+ 20973us) raft_consensus.cc:1529] Early marking committed up to index 5
0116 02:59:34.677364 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:34.691729 (+ 14365us) log.cc:844] Serialized 26 byte log entry
0116 02:59:35.059820 (+368091us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:35.059866 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:35.059953 (+ 87us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:36.239745 (+1179792us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:36.682986 (+443241us) raft_consensus.cc:1680] finished
0116 02:59:36.683012 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:36.717757 (+ 34745us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4479,"raft.queue_time_us":20592,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":15030,"spinlock_wait_cycles":43349888,"thread_start_us":312530,"threads_started":1,"wal-append.queue_time_us":502451}
I20250116 02:59:36.785341 20682 rpcz_store.cc:276] 0116 02:59:35.035323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.047344 (+ 12021us) service_pool.cc:225] Handling call
0116 02:59:35.063703 (+ 16359us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:36.589195 (+1525492us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.737711 (+148516us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b5400191ed4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:36.737915 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3239026432}
I20250116 02:59:36.787393 20753 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:36.733588 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.680882 22821 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.572499 23113 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.669852 23246 consensus_queue.cc:1035] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20250116 02:59:36.695439 23252 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:36.578486 23257 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.744133 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.794358 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:36.795468 23300 log.cc:926] Time spent T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20250116 02:59:36.747063 22868 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.750757 22998 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:36.797026 23257 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:36.797775 23257 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:36.798259 22998 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:36.486891 20675 rpcz_store.cc:276] 0116 02:59:34.101370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.115974 (+ 14604us) service_pool.cc:225] Handling call
0116 02:59:34.170936 (+ 54962us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:34.445527 (+274591us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.127916 (+1682389us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b54001c57d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:36.128251 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":133058944}
I20250116 02:59:36.799155 20755 raft_consensus.cc:2388] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:36.714211 23083 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s user 0.025s sys 0.039s
I20250116 02:59:36.800108 22843 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.742962 23255 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:36.801230 23255 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.803426 23255 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.803987 23255 consensus_queue.cc:1046] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250116 02:59:36.784166 23301 log.cc:926] Time spent T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.106s user 0.000s sys 0.002s
I20250116 02:59:36.806946 23294 raft_consensus.cc:3049] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:59:36.807374 23294 raft_consensus.cc:738] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:36.808938 23256 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:36.808532 23294 consensus_queue.cc:260] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.743109 23071 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:36.799070 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 469) took 1754 ms. Trace:
I20250116 02:59:36.812866 20748 rpcz_store.cc:276] 0116 02:59:35.044602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.060572 (+ 15970us) service_pool.cc:225] Handling call
0116 02:59:35.167356 (+106784us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:35.356505 (+189149us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:35.482153 (+125648us) log.cc:844] Serialized 14 byte log entry
0116 02:59:36.369840 (+887687us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:36.578813 (+208973us) log.cc:844] Serialized 26 byte log entry
0116 02:59:36.659155 (+ 80342us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:36.669643 (+ 10488us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.677707 (+ 8064us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:36.798819 (+121112us) raft_consensus.cc:1680] finished
0116 02:59:36.798842 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:36.799000 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21753,"prepare.queue_time_us":58528,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":42,"raft.queue_time_us":172489,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":26177,"thread_start_us":504253,"threads_started":1,"wal-append.queue_time_us":1132699}
I20250116 02:59:36.719831 23302 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.595883 23242 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.703814 23212 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:36.782083 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:36.712422 23271 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:36.693130 22936 raft_consensus.cc:738] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:36.791898 20608 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:36.794112 22871 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.798305 23257 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.764562 20757 raft_consensus.cc:1270] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250116 02:59:36.794467 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2067) took 1972 ms. Trace:
I20250116 02:59:36.792546 21300 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:36.824335 20605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a569f56f9d2143869b29c78321047d00"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43854
W20250116 02:59:36.781311 23303 log.cc:926] Time spent T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.104s user 0.001s sys 0.001s
W20250116 02:59:36.825842 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 116966us
I20250116 02:59:36.839885 20605 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:59:36.837918 20756 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:36.827755 20606 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:36.742748 23241 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:36.830308 20647 leader_election.cc:336] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.547s (SENT)
I20250116 02:59:36.682472 23209 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
W20250116 02:59:36.839414 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 46948.5us
I20250116 02:59:36.840310 20605 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:36.827584 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.838335 20607 rpcz_store.cc:276] 0116 02:59:34.822511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.822686 (+ 175us) service_pool.cc:225] Handling call
0116 02:59:34.823065 (+ 379us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:34.874505 (+ 51440us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:34.874535 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:34.902946 (+ 28411us) log.cc:844] Serialized 26 byte log entry
0116 02:59:35.025799 (+122853us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:35.025853 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:35.025935 (+ 82us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:35.877965 (+852030us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:36.793571 (+915606us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b2400048248. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323c2d9d4 00007fc323c2badb 00007fc323cba289 00007fc323cb9f15 00007fc323cb9470 00007fc3269b9f88 00007fc3269ba6e8 00007fc326991f98 00007fc32698d3aa 00007fc32698c7f6
0116 02:59:36.794270 (+ 699us) raft_consensus.cc:1680] finished
0116 02:59:36.794289 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:36.794411 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10691,"raft.queue_time_us":32505,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4047,"spinlock_wait_cycles":53905152,"thread_start_us":102635,"threads_started":1,"wal-append.queue_time_us":196280}
I20250116 02:59:36.858346 20647 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:36.861630 20750 raft_consensus.cc:2371] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5. Re-sending same reply.
I20250116 02:59:36.826009 23256 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.822585 22519 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:36.824904 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 545) took 1104 ms. Trace:
W20250116 02:59:36.840113 20572 leader_election.cc:336] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.074s (SENT)
W20250116 02:59:36.827628 20608 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 431) took 1355 ms (client timeout 1792 ms). Trace:
W20250116 02:59:36.872161 20608 rpcz_store.cc:269] 0116 02:59:35.472289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.549542 (+ 77253us) service_pool.cc:225] Handling call
0116 02:59:36.827557 (+1278015us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:36.826925 23302 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:36.790515 23205 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.071s
I20250116 02:59:36.826399 23271 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:36.870260 22936 consensus_queue.cc:260] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.870569 21327 rpcz_store.cc:276] 0116 02:59:35.720322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.781855 (+ 61533us) service_pool.cc:225] Handling call
0116 02:59:36.822781 (+1040926us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:36.859546 22998 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.921187 22998 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.924321 22998 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:36.925153 22998 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:36.925733 22998 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.928669 23209 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:36.934212 23113 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:36.935629 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:36.936702 20751 raft_consensus.cc:2363] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:36.937813 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.939131 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:36.821686 22821 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.939497 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1693) took 2063 ms. Trace:
I20250116 02:59:36.940089 20602 rpcz_store.cc:276] 0116 02:59:34.875758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.143309 (+267551us) service_pool.cc:225] Handling call
0116 02:59:35.460332 (+317023us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5800004898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:35.470016 (+ 9684us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:36.880469 (+1410453us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:36.880508 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:36.880559 (+ 51us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:36.880600 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.939169 (+ 58569us) spinlock_profiling.cc:243] Waited 898 ms on lock 0x7b5400109354. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:36.939245 (+ 76us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:36.939413 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":200470,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":12629,"spinlock_wait_cycles":2093229568,"thread_start_us":508529,"threads_started":1}
I20250116 02:59:36.886102 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.773821 21331 rpcz_store.cc:276] 0116 02:59:34.849655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.854548 (+ 4893us) service_pool.cc:225] Handling call
0116 02:59:36.127797 (+1273249us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:36.567410 (+439613us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24519936}
I20250116 02:59:36.859687 22868 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:36.942674 22868 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:36.943173 22868 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:36.894398 22936 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:36.878247 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:36.819276 23273 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.899883 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2084) took 1405 ms. Trace:
I20250116 02:59:36.907223 23063 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:36.944731 20606 rpcz_store.cc:276] 0116 02:59:35.494930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.699012 (+1204082us) service_pool.cc:225] Handling call
0116 02:59:36.827568 (+128556us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:36.840808 (+ 13240us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.899589 (+ 58781us) spinlock_profiling.cc:243] Waited 86.7 ms on lock 0x7b54000564d0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:36.899790 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199329024}
I20250116 02:59:36.907863 23242 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:36.910269 23257 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.946260 23273 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:36.841017 22871 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.884878 23241 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:36.947474 23241 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.948670 22871 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.915035 20674 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:36.916472 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.949623 20755 raft_consensus.cc:2463] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:36.950030 23241 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:36.916090 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.917419 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 546) took 1196 ms. Trace:
I20250116 02:59:36.875604 22519 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:36.920114 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 80043.3us
I20250116 02:59:36.870879 23256 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.945004 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.893635 23275 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:36.951581 20753 raft_consensus.cc:2371] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5. Re-sending same reply.
I20250116 02:59:36.935865 23248 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.952548 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:36.921799 23245 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:36.964423 20750 raft_consensus.cc:2388] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:36.886893 23271 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:36.969161 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.970079 20753 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:36.970818 22957 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.971618 22957 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.951359 23277 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:36.975225 23207 consensus_meta.cc:287] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.503s user 0.009s sys 0.001s
I20250116 02:59:36.975756 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.976317 20752 raft_consensus.cc:2405] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250116 02:59:36.976677 20750 raft_consensus.cc:2388] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:36.942150 20757 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
W20250116 02:59:36.977483 20752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1559) took 1837 ms (client timeout 1850 ms). Trace:
I20250116 02:59:36.946451 23257 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:36.951681 20572 leader_election.cc:336] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.733s (SENT)
I20250116 02:59:36.978852 20572 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:36.977903 20752 rpcz_store.cc:269] 0116 02:59:35.140229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.154482 (+ 14253us) service_pool.cc:225] Handling call
0116 02:59:36.977242 (+1822760us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b540033ccd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:36.977417 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4148853120}
I20250116 02:59:36.953590 20751 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:36.959407 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1575) took 1332 ms. Trace:
I20250116 02:59:36.982333 20756 rpcz_store.cc:276] 0116 02:59:35.627139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.627334 (+ 195us) service_pool.cc:225] Handling call
0116 02:59:36.959231 (+1331897us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b5400396f54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:36.959344 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1130188544}
I20250116 02:59:36.951035 20748 raft_consensus.cc:2388] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:36.953058 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 122773us
I20250116 02:59:36.969420 22930 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.942065 22821 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.975713 23277 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:36.995173 20720 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:36.978549 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.950763 21332 rpcz_store.cc:276] 0116 02:59:35.720613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:35.720911 (+ 298us) service_pool.cc:225] Handling call
0116 02:59:35.797906 (+ 76995us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:35.798226 (+ 320us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:35.798250 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:36.353256 (+555006us) log.cc:844] Serialized 26 byte log entry
0116 02:59:36.573319 (+220063us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:36.573365 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.573460 (+ 95us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:36.917145 (+343685us) raft_consensus.cc:1680] finished
0116 02:59:36.917171 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:36.917351 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":209167,"threads_started":1,"wal-append.queue_time_us":529348}
W20250116 02:59:37.011588 20647 leader_election.cc:336] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.746s (SENT)
I20250116 02:59:36.957492 20605 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:36.977681 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:36.977667 22519 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:36.978237 20674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 426) took 2514 ms (client timeout 2230 ms). Trace:
W20250116 02:59:37.038828 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.007s sys 0.020s
I20250116 02:59:36.977862 23107 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:36.975291 23256 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:36.974417 22957 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:37.076913 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.008s sys 0.021s
I20250116 02:59:36.964329 23266 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:37.071199 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 78076.3us
W20250116 02:59:37.072036 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 92451.7us
W20250116 02:59:37.083572 20647 leader_election.cc:336] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.874s (SENT)
I20250116 02:59:37.081694 20755 raft_consensus.cc:1270] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250116 02:59:37.039304 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:37.052927 23251 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.006s sys 0.037s
W20250116 02:59:37.092950 23251 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.801s user 0.007s sys 0.038s
W20250116 02:59:37.077530 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 255445us
I20250116 02:59:37.081130 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:37.033931 23071 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:37.100634 20757 raft_consensus.cc:2363] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6.
W20250116 02:59:37.078528 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 126902us
W20250116 02:59:37.103358 20572 leader_election.cc:336] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.211s (SENT)
W20250116 02:59:37.087266 20680 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.003s sys 0.015s
W20250116 02:59:37.104498 20680 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s user 0.003s sys 0.015s
I20250116 02:59:37.081197 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:37.090965 20752 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:37.076974 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 210657us
I20250116 02:59:37.032833 23207 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:37.092324 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 307691us
W20250116 02:59:37.038450 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.035s sys 0.086s
I20250116 02:59:37.082692 20605 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:37.091792 21299 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.008s sys 0.004s
I20250116 02:59:37.125512 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.092677 23256 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:37.134686 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.135754 21332 raft_consensus.cc:2405] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:37.136772 20720 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:37.138417 21299 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.012s sys 0.007s
W20250116 02:59:37.138720 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.655s user 0.038s sys 0.087s
I20250116 02:59:37.139277 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:37.140264 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:37.140877 20605 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 8
I20250116 02:59:37.141005 20608 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:37.139647 20751 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:37.144028 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:37.139292 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 631826us
W20250116 02:59:37.139842 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 656436us
I20250116 02:59:37.147994 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:37.160952 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:37.161186 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:37.162127 20602 raft_consensus.cc:2405] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250116 02:59:37.163877 20683 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20250116 02:59:37.165297 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 62084.5us
I20250116 02:59:37.169040 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:37.172292 21299 connection.cc:232] RPC call timeout handler was delayed by -1.0356021710000505s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
W20250116 02:59:37.173519 20572 leader_election.cc:336] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.744s (SENT)
I20250116 02:59:37.159502 20750 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
W20250116 02:59:37.174980 20572 leader_election.cc:336] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.850s (SENT)
I20250116 02:59:37.169968 20678 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
W20250116 02:59:37.176026 20572 leader_election.cc:336] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.869s (SENT)
I20250116 02:59:37.082202 23107 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:37.188232 20750 raft_consensus.cc:1235] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:37.126365 23063 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:37.182655 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:37.206635 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:37.206669 20606 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:37.192996 20683 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250116 02:59:37.209977 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 126427us
W20250116 02:59:37.210950 20647 leader_election.cc:336] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.917s (SENT)
I20250116 02:59:37.211367 20647 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:37.204180 20751 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:37.218828 20755 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250116 02:59:37.218984 20605 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:37.220310 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:37.222754 20756 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:37.207650 20604 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:37.136415 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:37.232702 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 131371us
W20250116 02:59:37.078364 20674 rpcz_store.cc:269] 0116 02:59:34.463805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:34.467230 (+ 3425us) service_pool.cc:225] Handling call
0116 02:59:36.958841 (+2491611us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54001c4dd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:36.978103 (+ 19262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4237225728}
I20250116 02:59:37.143883 20675 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:37.134037 23256 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.183243 23207 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:37.236654 23308 log.cc:926] Time spent T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20250116 02:59:37.224970 23265 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20250116 02:59:37.237531 20753 raft_consensus.cc:1270] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250116 02:59:37.254170 20753 pending_rounds.cc:77] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 1
I20250116 02:59:37.254455 20753 pending_rounds.cc:99] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250116 02:59:37.107165 22519 consensus_queue.cc:1046] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.118s
I20250116 02:59:37.263273 20720 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:37.239377 23266 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:37.218582 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:37.240392 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:37.244944 20750 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:37.224699 23063 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:37.241103 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:37.176677 22753 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:37.249743 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.226017 23071 raft_consensus.cc:2758] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250116 02:59:37.255333 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:37.278498 20755 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:37.254587 20571 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:37.259622 23256 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:37.260489 21299 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:37.260946 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.263679 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:37.300146 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 223918us
I20250116 02:59:37.303992 20606 raft_consensus.cc:2405] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:37.255424 20753 raft_consensus.cc:2883] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:37.212692 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:37.246557 23311 consensus_queue.cc:1035] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250116 02:59:37.270751 20752 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:37.274384 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:37.276618 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:37.305503 20604 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:37.305684 20605 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:37.240156 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:37.292330 20571 connection.cc:232] RPC call timeout handler was delayed by -1.7213605440000492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
I20250116 02:59:37.250622 22998 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:37.245519 22994 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:37.305830 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 94936.9us
I20250116 02:59:37.286772 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:37.256817 20675 raft_consensus.cc:604] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Received request to transfer leadership to TS 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:37.310590 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 69575.8us
I20250116 02:59:37.314527 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:37.312247 20571 connection.cc:232] RPC call timeout handler was delayed by -1.2443217560000903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
W20250116 02:59:37.314302 20647 leader_election.cc:336] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.924s (SENT)
I20250116 02:59:37.315618 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:37.317260 20680 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:37.324784 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:37.323844 21300 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:37.325759 20606 raft_consensus.cc:2388] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:37.315888 20678 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:37.273967 23063 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:37.282749 22753 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.309926 23235 consensus_queue.cc:1046] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20250116 02:59:37.303646 22957 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:37.336637 22753 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:37.337003 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:37.326516 23255 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:37.327044 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:37.338097 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:37.338600 20681 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:37.326347 23241 consensus_queue.cc:1046] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20250116 02:59:37.337145 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:37.340948 20679 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:37.342552 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:37.329762 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 136308us
I20250116 02:59:37.328423 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:37.344096 20572 leader_election.cc:336] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.995s (SENT)
I20250116 02:59:37.344538 20572 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:37.345458 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:37.346822 20682 raft_consensus.cc:3049] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:37.347178 20682 raft_consensus.cc:738] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:37.271297 23321 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:37.347345 20647 leader_election.cc:336] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.961s (SENT)
I20250116 02:59:37.347926 20647 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:37.343331 20680 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:37.326159 20675 raft_consensus.cc:612] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:37.274444 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:37.351958 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 88810.9us
I20250116 02:59:37.353789 20571 leader_election.cc:400] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:37.324397 21327 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:37.309966 23207 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:37.309746 23322 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:37.256399 23313 consensus_queue.cc:1035] T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
I20250116 02:59:37.371549 20571 leader_election.cc:403] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20250116 02:59:37.371472 20647 leader_election.cc:336] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.024s (SENT)
I20250116 02:59:37.389998 20647 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:37.352094 20682 consensus_queue.cc:260] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:37.339265 23266 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:59:37.385568 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.391348 20682 raft_consensus.cc:3054] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:37.391798 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 478) took 1065 ms. Trace:
I20250116 02:59:37.386230 20756 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:37.358245 22957 raft_consensus.cc:695] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:37.411752 21332 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:37.412318 20720 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:37.406999 20754 rpcz_store.cc:276] 0116 02:59:36.326672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.326837 (+ 165us) service_pool.cc:225] Handling call
0116 02:59:36.327187 (+ 350us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:36.362904 (+ 35717us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:36.362940 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:36.363534 (+ 594us) log.cc:844] Serialized 26 byte log entry
0116 02:59:36.648124 (+284590us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:36.648165 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:36.648240 (+ 75us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:37.391510 (+743270us) raft_consensus.cc:1680] finished
0116 02:59:37.391533 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:37.391716 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31769,"raft.queue_time_us":36099,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":74,"thread_start_us":284125,"threads_started":1,"wal-append.queue_time_us":601876}
I20250116 02:59:37.278373 22908 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.093s
I20250116 02:59:37.309710 23246 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20250116 02:59:37.427315 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:37.429246 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 80235.8us
I20250116 02:59:37.355448 23248 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:37.435680 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.443105 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.440722 21334 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:37.387796 22936 raft_consensus.cc:2798] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:37.337898 23255 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:37.449687 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 134555us
I20250116 02:59:37.348197 23302 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.078s
I20250116 02:59:37.393101 23322 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:37.448938 20681 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
W20250116 02:59:37.474248 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 68984.2us
W20250116 02:59:37.457084 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 118328us
I20250116 02:59:37.336621 22994 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.483466 20649 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:37.480098 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 108645us
W20250116 02:59:37.484467 20647 leader_election.cc:336] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.200s (SENT)
I20250116 02:59:37.480069 20682 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:37.435632 23310 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:37.485702 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 73473us
I20250116 02:59:37.482391 23320 consensus_queue.cc:1046] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20250116 02:59:37.456269 22998 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:37.489867 20647 leader_election.cc:336] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.721s (SENT)
I20250116 02:59:37.479053 23322 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:37.410996 22957 consensus_queue.cc:237] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:37.494738 20649 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:37.495874 20647 leader_election.cc:336] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.865s (SENT)
I20250116 02:59:37.495806 20649 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:37.496368 20647 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:37.487208 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:37.497262 20649 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:37.497936 20681 raft_consensus.cc:604] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:37.453830 22936 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:37.502066 20720 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:37.451848 23252 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:37.473909 23266 raft_consensus.cc:695] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:37.479724 23207 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:37.530639 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2089) took 1067 ms. Trace:
I20250116 02:59:37.503404 22791 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election won for term 8
W20250116 02:59:37.529067 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 46264.7us
I20250116 02:59:37.538064 20649 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:37.539521 21336 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:37.479564 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.545372 20601 rpcz_store.cc:276] 0116 02:59:36.463410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.740127 (+276717us) service_pool.cc:225] Handling call
0116 02:59:36.793662 (+ 53535us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:36.811265 (+ 17603us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:37.078300 (+267035us) log.cc:844] Serialized 14 byte log entry
0116 02:59:37.252431 (+174131us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:37.252486 (+ 55us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:37.252532 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:37.530264 (+277732us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b540014e1d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:37.530349 (+ 85us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:37.530533 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":351564,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":13695,"raft.queue_time_us":1994,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":16973,"spinlock_wait_cycles":39509760,"thread_start_us":416266,"threads_started":2,"wal-append.queue_time_us":362315}
I20250116 02:59:37.454291 23248 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.498879 23317 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:37.538259 22936 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:37.537748 23256 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:37.543304 22998 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:37.528555 22519 raft_consensus.cc:2743] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:37.539927 22930 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20250116 02:59:37.588707 23265 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:37.589342 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:37.552207 23320 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:37.602010 20681 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:37.570029 23317 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:37.587221 22871 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:37.590343 22908 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:37.591702 23171 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:37.590466 23256 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:37.606284 23324 log.cc:926] Time spent T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.355s user 0.001s sys 0.000s
I20250116 02:59:37.642300 20755 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:37.606105 23320 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:37.647099 23320 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:37.649499 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.638345 22908 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.653645 22908 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:37.641436 23256 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:37.639029 23265 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.667476 23265 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:37.668874 23271 consensus_meta.cc:287] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.673s user 0.007s sys 0.003s
I20250116 02:59:37.669412 23271 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.662161 23320 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:37.675009 23248 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:37.675902 23294 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:59:37.676618 23063 raft_consensus.cc:3049] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:37.677031 23063 raft_consensus.cc:738] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
W20250116 02:59:37.640542 22868 consensus_meta.cc:287] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.697s user 0.007s sys 0.007s
W20250116 02:59:37.678961 20647 leader_election.cc:336] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.896s (SENT)
I20250116 02:59:37.679272 22868 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.679733 20647 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:37.679090 22753 raft_consensus.cc:3049] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:37.680760 22753 raft_consensus.cc:738] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:37.681206 23338 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:37.682013 22868 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:37.679616 23257 raft_consensus.cc:3049] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:37.682860 23257 raft_consensus.cc:738] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:37.683746 23257 consensus_queue.cc:260] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.685177 23257 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:37.686292 23320 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:37.682327 22753 consensus_queue.cc:260] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.691390 22994 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:37.678062 23063 consensus_queue.cc:260] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:37.700147 23245 consensus_meta.cc:287] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.719s user 0.010s sys 0.001s
I20250116 02:59:37.700795 23251 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:37.701361 23251 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.702785 21329 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:37.706907 23327 log.cc:926] Time spent T 1d7658a3a42c44f485085e75a4bd7b88 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.348s user 0.000s sys 0.004s
W20250116 02:59:37.651049 23318 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.207s user 0.001s sys 0.000s
W20250116 02:59:37.718559 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 148017us
I20250116 02:59:37.635727 23317 raft_consensus.cc:3054] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:37.747885 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 245929us
I20250116 02:59:37.749428 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:37.750123 20607 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:37.751994 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 198549us
I20250116 02:59:37.635535 23277 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:37.753352 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 311086us
I20250116 02:59:37.759188 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:37.716738 23063 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:37.762526 20571 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:37.763537 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:37.765311 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:37.766083 20606 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:37.768682 21300 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:37.774190 20647 leader_election.cc:336] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.791s (SENT)
I20250116 02:59:37.774837 20647 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:37.761461 23248 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:37.690253 23266 consensus_queue.cc:237] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:37.794405 20647 leader_election.cc:336] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.048s (SENT)
I20250116 02:59:37.782023 23071 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:37.793185 20571 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:37.795529 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 451438us
I20250116 02:59:37.814745 20608 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:37.781410 22868 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:37.821704 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:37.822489 20679 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:37.717940 23322 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:37.789350 23299 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:37.823756 20572 leader_election.cc:336] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.201s (SENT)
I20250116 02:59:37.808197 20682 raft_consensus.cc:1235] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250116 02:59:37.789117 23255 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:37.791361 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.834594 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.835132 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2083) took 2341 ms. Trace:
I20250116 02:59:37.851943 20600 rpcz_store.cc:276] 0116 02:59:35.494494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.685033 (+1190539us) service_pool.cc:225] Handling call
0116 02:59:36.711423 (+ 26390us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:36.797833 (+ 86410us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:36.797869 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:36.798437 (+ 568us) log.cc:844] Serialized 26 byte log entry
0116 02:59:37.094676 (+296239us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:37.094728 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:37.219577 (+124849us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x7b5400093154. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:37.219640 (+ 63us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:37.822478 (+602838us) raft_consensus.cc:1680] finished
0116 02:59:37.822503 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:37.835037 (+ 12534us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6815,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":119,"spinlock_wait_cycles":181180160,"thread_start_us":295698,"threads_started":1,"wal-append.queue_time_us":510508}
I20250116 02:59:37.788205 22791 raft_consensus.cc:695] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:37.841645 20572 leader_election.cc:336] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.223s (SENT)
I20250116 02:59:37.844260 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:37.972388 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:37.848197 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:37.834201 20571 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:37.982831 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:37.849854 21333 raft_consensus.cc:2388] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:38.046766 21334 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:37.820143 23071 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:37.921716 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:37.838850 22868 raft_consensus.cc:695] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:38.030035 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 173456us
I20250116 02:59:37.870963 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:38.046227 20678 raft_consensus.cc:2388] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:38.055546 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:38.056070 20681 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:37.836298 23311 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:38.048501 20646 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:38.049692 20755 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
W20250116 02:59:37.963218 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 111143us
I20250116 02:59:37.850766 20675 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:38.047072 20754 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:38.029914 20647 leader_election.cc:336] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.621s (SENT)
I20250116 02:59:38.045718 20572 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:38.050884 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 216758us
I20250116 02:59:37.846959 23107 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:38.064539 20647 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:38.065026 20571 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:37.845198 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 75210.1us
W20250116 02:59:38.111577 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 81718.4us
I20250116 02:59:38.027029 23271 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:38.019784 23255 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:38.121999 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 57075.6us
I20250116 02:59:38.088148 22930 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20250116 02:59:38.072644 23275 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250116 02:59:38.111246 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 44783.5us
I20250116 02:59:37.867033 21331 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:38.074889 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:38.120324 20720 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:38.054042 23071 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:38.122437 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 280666us
W20250116 02:59:38.116984 22936 consensus_meta.cc:287] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.513s user 0.006s sys 0.006s
I20250116 02:59:38.132606 21330 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:38.060762 23355 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:38.113850 20675 raft_consensus.cc:604] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:38.131702 20606 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:38.113370 22957 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:38.137653 20755 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:38.118933 21327 consensus_meta.cc:287] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.680s user 0.009s sys 0.009s
I20250116 02:59:38.114313 23320 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:38.149148 20572 leader_election.cc:336] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.549s (SENT)
I20250116 02:59:38.125352 23248 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250116 02:59:38.187469 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 67245.9us
I20250116 02:59:38.154788 23355 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:38.136297 23275 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:38.160351 20649 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:38.160199 20681 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:38.180378 20755 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:38.157598 23107 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:38.203243 22821 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:38.203866 22821 raft_consensus.cc:695] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:38.185874 23322 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:38.189208 20754 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:38.191334 20646 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:38.194319 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 49942.1us
I20250116 02:59:38.159394 20675 raft_consensus.cc:612] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:38.208047 22821 consensus_queue.cc:237] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:38.210419 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 87102.9us
I20250116 02:59:38.136462 23255 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:38.187556 23248 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:38.199113 23355 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:38.219733 20571 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:38.220819 20605 raft_consensus.cc:1270] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:38.221486 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:38.232928 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:38.217883 23358 log.cc:926] Time spent T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20250116 02:59:38.233544 20607 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:38.205869 23275 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:38.222950 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:38.189679 21327 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:38.226513 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:38.241959 20720 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:38.227582 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 82711.8us
I20250116 02:59:38.229467 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:38.150007 22936 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:38.222237 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:38.247082 20602 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:38.225558 22957 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:38.154852 22868 consensus_queue.cc:237] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:38.236336 23248 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:38.249616 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 87040.5us
I20250116 02:59:38.148125 22791 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:38.250449 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 59254us
I20250116 02:59:38.251926 21300 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:38.243073 20752 raft_consensus.cc:2949] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:38.240177 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 852) took 1135 ms. Trace:
I20250116 02:59:38.265987 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:38.266728 20676 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:38.274322 20602 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:38.252907 23107 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:38.282322 20678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:38.291486 21300 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:38.261057 23356 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:38.294116 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:38.298090 23317 consensus_meta.cc:287] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.536s user 0.009s sys 0.001s
I20250116 02:59:38.298724 23317 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:38.301353 23317 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:38.304438 20720 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:38.305989 20720 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:38.308178 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:38.310601 23255 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:38.310782 23356 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:38.311266 23356 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:38.312361 23366 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20250116 02:59:38.259621 23363 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:38.262905 23313 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250116 02:59:38.313616 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 80754.4us
I20250116 02:59:38.268370 23364 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:38.319248 20571 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:38.259457 22957 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:38.294230 21331 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:38.328241 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:38.338717 20605 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
W20250116 02:59:38.350662 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 44777.7us
I20250116 02:59:38.341228 20649 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:38.353047 20720 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:38.350349 21300 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:38.336522 20607 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:38.265398 21327 rpcz_store.cc:276] 0116 02:59:37.104809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.105990 (+ 1181us) service_pool.cc:225] Handling call
0116 02:59:38.240093 (+1134103us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:38.326947 23256 consensus_meta.cc:287] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.663s user 0.011s sys 0.001s
W20250116 02:59:38.328401 23257 consensus_meta.cc:287] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.643s user 0.006s sys 0.006s
I20250116 02:59:38.363474 20649 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:38.321519 23363 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:38.323148 23313 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:38.338986 23294 consensus_meta.cc:287] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.663s user 0.012s sys 0.001s
I20250116 02:59:38.367208 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:38.349684 23337 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:38.326926 23348 raft_consensus.cc:2758] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20250116 02:59:38.327806 23063 raft_consensus.cc:2743] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:38.339221 23107 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:38.384450 20678 raft_consensus.cc:604] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:38.391345 20647 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.561s (SENT) [suppressed 43 similar messages]
I20250116 02:59:38.404109 23369 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:38.383579 23351 raft_consensus.cc:2798] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:38.407749 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 552) took 1018 ms. Trace:
I20250116 02:59:38.363971 23317 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:38.393132 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 855) took 1232 ms. Trace:
I20250116 02:59:38.415100 20678 raft_consensus.cc:612] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250116 02:59:38.394565 23333 log.cc:926] Time spent T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.752s user 0.006s sys 0.029s
I20250116 02:59:38.417086 21333 rpcz_store.cc:276] 0116 02:59:37.389641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.390652 (+ 1011us) service_pool.cc:225] Handling call
0116 02:59:38.407493 (+1016841us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b5400611a54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:38.407674 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":931146112}
I20250116 02:59:38.422462 23337 raft_consensus.cc:695] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:38.372340 23352 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:38.410953 21328 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 7. Ops: []
W20250116 02:59:38.422487 23318 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250116 02:59:38.375928 23313 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:38.440032 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 87071.1us
I20250116 02:59:38.421101 23317 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:38.433609 23348 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:38.426546 22791 consensus_queue.cc:1046] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20250116 02:59:38.434553 23369 raft_consensus.cc:695] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:38.469162 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 149999us
I20250116 02:59:38.421319 23107 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:38.375435 23256 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:38.444046 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1641) took 1159 ms. Trace:
I20250116 02:59:38.477335 20677 rpcz_store.cc:276] 0116 02:59:37.284660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.310213 (+ 25553us) service_pool.cc:225] Handling call
0116 02:59:37.310633 (+ 420us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:37.336874 (+ 26241us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:37.336933 (+ 59us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:37.337540 (+ 607us) log.cc:844] Serialized 26 byte log entry
0116 02:59:37.498094 (+160554us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:37.498143 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:37.498233 (+ 90us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:38.393730 (+895497us) spinlock_profiling.cc:243] Waited 69.5 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:38.429451 (+ 35721us) raft_consensus.cc:1680] finished
0116 02:59:38.429476 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:38.443950 (+ 14474us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5776,"raft.queue_time_us":13211,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":126,"spinlock_wait_cycles":159930624,"thread_start_us":159791,"threads_started":1,"wal-append.queue_time_us":216496}
I20250116 02:59:38.375501 23363 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:38.432057 23351 raft_consensus.cc:695] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:38.445658 20720 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:38.470199 20571 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:38.450436 23317 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:38.443893 23352 raft_consensus.cc:695] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:38.507139 20571 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:38.457649 22936 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:38.422392 21329 rpcz_store.cc:276] 0116 02:59:37.160835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.194485 (+ 33650us) service_pool.cc:225] Handling call
0116 02:59:38.392802 (+1198317us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b5400520dd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:38.393031 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":751012608}
I20250116 02:59:38.546696 20679 raft_consensus.cc:1235] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20250116 02:59:38.488729 23325 log.cc:926] Time spent T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.822s user 0.011s sys 0.004s
W20250116 02:59:38.505502 23372 log.cc:926] Time spent T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.094s user 0.002s sys 0.000s
I20250116 02:59:38.452303 23063 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.088s
I20250116 02:59:38.550575 23368 raft_consensus.cc:2798] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:38.550350 23369 consensus_queue.cc:237] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:38.552415 23331 log.cc:926] Time spent T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 1.068s user 0.003s sys 0.011s
I20250116 02:59:38.558079 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1604) took 1639 ms. Trace:
W20250116 02:59:38.568316 23375 log.cc:926] Time spent T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20250116 02:59:38.568941 23337 consensus_queue.cc:237] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:38.570115 23257 raft_consensus.cc:3049] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:38.563913 23311 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:38.570685 23377 consensus_queue.cc:1046] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 02:59:38.571916 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 547) took 1200 ms. Trace:
I20250116 02:59:38.572255 23379 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:38.595943 23343 log.cc:926] Time spent T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.957s user 0.004s sys 0.011s
I20250116 02:59:38.598984 20753 rpcz_store.cc:276] 0116 02:59:36.953378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.970871 (+ 17493us) service_pool.cc:225] Handling call
0116 02:59:36.978243 (+ 7372us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:37.313657 (+335414us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:37.539154 (+225497us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b540033ccd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:38.557789 (+1018635us) spinlock_profiling.cc:243] Waited 6.99 ms on lock 0x7b540033ccd0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:38.557995 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":611243648}
I20250116 02:59:38.622062 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:38.622771 23256 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:38.627844 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 182348us
I20250116 02:59:38.641106 20646 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:38.653096 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 261768us
I20250116 02:59:38.677654 20676 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:38.681154 21299 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:38.624228 23257 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:38.627262 23311 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:38.776588 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 292667us
I20250116 02:59:38.692981 22993 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:38.653610 21328 rpcz_store.cc:276] 0116 02:59:37.371483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.376169 (+ 4686us) service_pool.cc:225] Handling call
0116 02:59:38.410730 (+1034561us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:38.440866 (+ 30136us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:38.571626 (+130760us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b5400611a50. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:38.571831 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2133521280}
I20250116 02:59:38.781756 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:38.788733 20753 raft_consensus.cc:2363] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:38.683076 22957 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:38.761158 23228 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:38.763835 23311 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:38.854557 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:38.859897 21330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 860) took 1174 ms. Trace:
I20250116 02:59:38.860404 21330 rpcz_store.cc:276] 0116 02:59:37.685912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.692042 (+ 6130us) service_pool.cc:225] Handling call
0116 02:59:37.693153 (+ 1111us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:38.159825 (+466672us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:38.623970 (+464145us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b540060f4d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:38.859811 (+235841us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":987165568}
W20250116 02:59:38.862757 23378 log.cc:926] Time spent T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20250116 02:59:38.867612 20676 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250116 02:59:38.868137 23071 consensus_meta.cc:287] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.711s user 0.019s sys 0.000s
I20250116 02:59:38.868592 23071 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:38.821805 22993 raft_consensus.cc:695] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:38.875022 23352 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:38.876153 22993 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:38.878803 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:38.885838 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:38.886708 20751 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:38.896432 20753 raft_consensus.cc:2363] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:38.877945 20674 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
W20250116 02:59:38.887363 23325 log.cc:926] Time spent T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20250116 02:59:38.839277 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1605) took 1867 ms. Trace:
I20250116 02:59:38.875023 23071 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:38.921270 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:38.781602 23368 raft_consensus.cc:695] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:38.914892 22957 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:38.996302 22957 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:38.997879 23251 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:38.999321 23275 consensus_meta.cc:287] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.750s user 0.012s sys 0.011s
I20250116 02:59:38.999920 23275 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:39.003242 23275 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:38.975349 20753 raft_consensus.cc:1270] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:38.982841 23256 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:38.847496 23228 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:38.905956 23351 consensus_queue.cc:237] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:38.927188 20748 rpcz_store.cc:276] 0116 02:59:36.971566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.072931 (+101365us) service_pool.cc:225] Handling call
0116 02:59:37.112560 (+ 39629us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:37.196669 (+ 84109us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:37.221254 (+ 24585us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:37.273511 (+ 52257us) log.cc:844] Serialized 26 byte log entry
0116 02:59:37.372246 (+ 98735us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:37.662863 (+290617us) spinlock_profiling.cc:243] Waited 891 us on lock 0x7b180037cd80. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc32699a08c 00007fc323db8bfc 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd3305 00007fc32a6e4107 00007fc326d198f7 00007fc326d2900f
0116 02:59:37.663013 (+ 150us) log.cc:844] Serialized 14 byte log entry
0116 02:59:37.663248 (+ 235us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:37.663332 (+ 84us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:37.688662 (+ 25330us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b180037cd80. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd67e8 00007fc323dd9cc1 00007fc323dd9c79 00007fc323dd9c09 00007fc323dd9bd1 00007fc323dd8ecd 00007fc32ce13234 00007fc32ce13069 00007fc323db8fe6
0116 02:59:38.551550 (+862888us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:38.782885 (+231335us) raft_consensus.cc:1680] finished
0116 02:59:38.788248 (+ 5363us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:38.821829 (+ 33581us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":125635,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":6782,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":36642,"spinlock_wait_cycles":16444672,"thread_start_us":125510,"threads_started":2,"wal-append.queue_time_us":81199}
I20250116 02:59:38.927143 23357 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:38.988211 23248 consensus_meta.cc:287] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.722s user 0.013s sys 0.008s
I20250116 02:59:38.989570 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 488) took 1809 ms. Trace:
I20250116 02:59:38.963578 23107 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:39.018323 20751 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:39.045740 23248 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:39.047495 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:39.048307 20600 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:39.048589 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:39.049211 20676 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
W20250116 02:59:39.050217 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 409203us
I20250116 02:59:38.853367 23257 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:39.051324 20599 raft_consensus.cc:1270] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250116 02:59:39.053256 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:39.054143 21333 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:39.032482 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:39.027642 20647 leader_election.cc:336] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 4.360s (SENT)
W20250116 02:59:39.026964 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 458874us
I20250116 02:59:39.032196 20757 rpcz_store.cc:276] 0116 02:59:37.180392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.181181 (+ 789us) service_pool.cc:225] Handling call
0116 02:59:37.181845 (+ 664us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:37.214308 (+ 32463us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:37.214342 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:37.242676 (+ 28334us) log.cc:844] Serialized 52 byte log entry
0116 02:59:37.535990 (+293314us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:37.536040 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:37.536126 (+ 86us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:37.673083 (+136957us) spinlock_profiling.cc:243] Waited 5.71 ms on lock 0x7b4800000e58. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32bbd35a1 00007fc32bbca687 00007fc32bbca51a 00007fc32bcbe9df 00007fc32bcbe6a7 00007fc32bcd2bac 00007fc32bcd2b58 00007fc32bcd2ad8 00007fc32bcd2a93 00007fc32bcd1d3c 00007fc32ac3ed16
0116 02:59:38.594926 (+921843us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:38.989320 (+394394us) raft_consensus.cc:1680] finished
0116 02:59:38.989349 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:38.989506 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4309,"raft.queue_time_us":36843,"raft.run_cpu_time_us":3872,"raft.run_wall_time_us":446918,"spinlock_wait_cycles":172699264,"thread_start_us":255038,"threads_started":1,"wal-append.queue_time_us":287415}
I20250116 02:59:39.070698 20676 raft_consensus.cc:1270] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250116 02:59:39.072165 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:39.073768 21333 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:39.080500 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:39.082170 20681 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:39.090276 20757 raft_consensus.cc:1212] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-5.6] Dedup: 5.6->[]
I20250116 02:59:39.091044 20757 raft_consensus.cc:1235] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: [2.4-5.6]
I20250116 02:59:39.050103 23335 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:39.096123 21333 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20250116 02:59:39.111009 23340 log.cc:926] Time spent T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 1.469s user 0.018s sys 0.014s
I20250116 02:59:39.113044 23366 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:39.088337 23071 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:39.114668 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 858) took 1844 ms. Trace:
W20250116 02:59:39.116089 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 88513.2us
W20250116 02:59:39.098906 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 43851.8us
I20250116 02:59:39.108799 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.117298 20757 raft_consensus.cc:2371] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5. Re-sending same reply.
I20250116 02:59:39.091157 20646 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:39.115131 23071 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:39.048215 23248 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:39.115149 21336 rpcz_store.cc:276] 0116 02:59:37.269959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.270197 (+ 238us) service_pool.cc:225] Handling call
0116 02:59:37.313295 (+ 43098us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:37.313518 (+ 223us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:37.554306 (+240788us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:39.114344 (+1560038us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b540058f054. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:39.114561 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":503235840}
I20250116 02:59:39.097561 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.126988 20681 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:39.135082 23071 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:39.136386 20748 raft_consensus.cc:2363] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:39.022492 23256 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:39.120095 23355 consensus_meta.cc:287] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.881s user 0.015s sys 0.000s
I20250116 02:59:39.144255 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:39.107605 23335 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:39.129376 23257 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:39.152473 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.152976 20605 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 02:59:39.038398 23357 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:39.157861 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1653) took 1862 ms. Trace:
I20250116 02:59:39.158574 20752 raft_consensus.cc:1235] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:39.143560 23071 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:39.174952 20683 rpcz_store.cc:276] 0116 02:59:37.294991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.341741 (+ 46750us) service_pool.cc:225] Handling call
0116 02:59:37.342070 (+ 329us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:37.388459 (+ 46389us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:37.388497 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:37.434097 (+ 45600us) log.cc:844] Serialized 26 byte log entry
0116 02:59:37.527316 (+ 93219us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:37.527359 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:37.527461 (+ 102us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:39.097786 (+1570325us) spinlock_profiling.cc:243] Waited 70.4 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:39.154679 (+ 56893us) raft_consensus.cc:1680] finished
0116 02:59:39.154703 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:39.157776 (+ 3073us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27076,"raft.queue_time_us":23365,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":48665,"spinlock_wait_cycles":161844096,"thread_start_us":92696,"threads_started":1,"wal-append.queue_time_us":165619}
W20250116 02:59:39.178865 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 161992us
I20250116 02:59:39.173542 22871 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250116 02:59:39.178411 23356 consensus_meta.cc:287] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.867s user 0.009s sys 0.005s
I20250116 02:59:39.216495 21300 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:39.161738 23335 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:39.143702 23338 consensus_queue.cc:1035] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20250116 02:59:39.196521 21329 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:39.264014 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:39.264789 21333 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:39.215152 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:39.196731 22957 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:39.275485 23389 log.cc:926] Time spent T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20250116 02:59:39.278111 23228 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:39.224563 23356 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:39.155754 23368 consensus_queue.cc:237] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:39.150768 23355 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:39.290594 21333 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:39.250656 23205 raft_consensus.cc:2798] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:39.196733 23071 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:39.226254 23273 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.083s
I20250116 02:59:39.307374 23107 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:39.326517 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:39.336300 20604 raft_consensus.cc:1235] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
W20250116 02:59:39.339310 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 320794us
I20250116 02:59:39.308754 20748 raft_consensus.cc:2371] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4. Re-sending same reply.
W20250116 02:59:39.348011 23323 log.cc:926] Time spent T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 1.712s user 0.013s sys 0.024s
W20250116 02:59:39.351233 23382 log.cc:926] Time spent T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20250116 02:59:39.350534 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 479) took 3022 ms. Trace:
I20250116 02:59:39.357946 20749 rpcz_store.cc:276] 0116 02:59:36.328046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.500730 (+172684us) service_pool.cc:225] Handling call
0116 02:59:36.501052 (+ 322us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:36.820621 (+319569us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:36.820658 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:36.825124 (+ 4466us) log.cc:844] Serialized 26 byte log entry
0116 02:59:37.141425 (+316301us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:37.141473 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:37.180922 (+ 39449us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b5400184cd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:37.180979 (+ 57us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:39.347295 (+2166316us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:39.350244 (+ 2949us) raft_consensus.cc:1680] finished
0116 02:59:39.350272 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:39.350450 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":734,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":4326,"spinlock_wait_cycles":891749888,"thread_start_us":266246,"threads_started":1,"wal-append.queue_time_us":596994}
I20250116 02:59:39.360374 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.361212 20749 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:39.417057 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.418319 20749 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:39.438159 20600 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
W20250116 02:59:39.439057 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 347996us
I20250116 02:59:39.439818 20646 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:39.464927 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:39.351958 23336 log.cc:926] Time spent T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 1.711s user 0.009s sys 0.012s
I20250116 02:59:39.482308 20749 raft_consensus.cc:2363] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:39.475710 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48834 (request call id 29) took 1209 ms. Trace:
I20250116 02:59:39.484975 20678 rpcz_store.cc:276] 0116 02:59:38.266150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:38.266308 (+ 158us) service_pool.cc:225] Handling call
0116 02:59:39.475533 (+1209225us) spinlock_profiling.cc:243] Waited 97.2 ms on lock 0x7b54001c5cd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d09bc3 00007fc32bc7082a 00007fc325349d14 00007fc325349ca2 00007fc325349bd1 00007fc325349b4c 00007fc325348e02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:39.475649 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":223529856}
I20250116 02:59:39.504298 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.447093 23205 raft_consensus.cc:695] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:39.447613 23390 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:39.504243 23321 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20250116 02:59:39.472323 22871 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:39.466306 23275 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:39.504243 23363 consensus_meta.cc:287] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.959s user 0.011s sys 0.007s
I20250116 02:59:39.523758 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.502430 23294 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:39.502877 23256 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:39.549171 23294 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:39.552639 20678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:39.553488 20678 raft_consensus.cc:604] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:59:39.553946 20678 raft_consensus.cc:612] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250116 02:59:39.557143 23394 log.cc:926] Time spent T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.331s user 0.001s sys 0.000s
I20250116 02:59:39.564291 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.566056 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.567518 23256 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:39.568305 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.568230 23256 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:39.569234 20753 raft_consensus.cc:2388] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 3: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:39.570312 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.346099 23107 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:39.744048 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20250116 02:59:39.750597 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 284920us
I20250116 02:59:39.608464 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:39.575354 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.572587 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:39.784431 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:39.784641 20571 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:39.785434 20678 raft_consensus.cc:2405] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250116 02:59:39.617229 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:39.788228 20608 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:39.782078 20751 raft_consensus.cc:2388] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:39.792039 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:39.794463 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.795351 20749 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
W20250116 02:59:39.677115 20647 leader_election.cc:336] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 6.375s (SENT)
W20250116 02:59:39.810813 20723 leader_election.cc:336] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 2.221s (SENT)
I20250116 02:59:39.612823 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 433) took 1394 ms. Trace:
I20250116 02:59:39.812004 20604 rpcz_store.cc:276] 0116 02:59:38.218637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:38.218771 (+ 134us) service_pool.cc:225] Handling call
0116 02:59:38.239878 (+ 21107us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:39.338441 (+1098563us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:39.612475 (+274034us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002e59d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:39.612733 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2520775552}
I20250116 02:59:39.613502 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:39.814780 20750 raft_consensus.cc:2371] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5. Re-sending same reply.
I20250116 02:59:39.815939 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.840538 20647 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:39.843618 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.555193 22871 raft_consensus.cc:695] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:39.525523 23273 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.338s
I20250116 02:59:39.736408 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.971616 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:39.975040 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 298014us
I20250116 02:59:39.984326 20678 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:39.525969 23390 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:39.816787 20750 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:39.878868 20749 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:39.999402 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.863529 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.876619 20649 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:40.010336 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.021160 20753 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:39.863401 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:39.989375 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:39.987095 23356 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:39.850531 23294 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:40.051003 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:40.096356 23311 consensus_meta.cc:287] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 1.245s user 0.009s sys 0.003s
I20250116 02:59:40.015900 23357 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:40.051980 20678 raft_consensus.cc:1235] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:39.997506 20677 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20250116 02:59:40.053191 23395 log.cc:926] Time spent T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.614s user 0.002s sys 0.000s
I20250116 02:59:39.996726 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:40.054327 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:40.050583 23364 consensus_meta.cc:287] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 1.701s user 0.014s sys 0.008s
I20250116 02:59:40.056494 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.059226 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:40.085467 20674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1640) took 2801 ms (client timeout 1794 ms). Trace:
I20250116 02:59:39.912150 22957 raft_consensus.cc:3049] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:40.063071 23245 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:39.772248 23335 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:39.958833 23412 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:39.957285 23071 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:39.848713 23248 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:39.727499 23363 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:39.829618 23369 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 02:59:40.084667 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 208165us
I20250116 02:59:40.128134 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.088938 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.084185 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:40.130916 20754 raft_consensus.cc:1235] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:40.295029 20608 raft_consensus.cc:2388] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:40.295214 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1983) took 1110 ms. Trace:
I20250116 02:59:40.083669 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:40.294564 20599 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:40.094664 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.083261 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.078730 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:59:40.297969 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:59:40.079685 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 461) took 2237 ms. Trace:
I20250116 02:59:40.302876 20679 rpcz_store.cc:276] 0116 02:59:37.841804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.841925 (+ 121us) service_pool.cc:225] Handling call
0116 02:59:37.846405 (+ 4480us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:38.547365 (+700960us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.079307 (+1531942us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b540016dfd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:40.079578 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1609656192}
I20250116 02:59:40.086051 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.081068 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:39.901580 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 117014us
I20250116 02:59:40.304901 20604 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 6
I20250116 02:59:40.294529 20605 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:40.000608 20752 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:40.074586 20504 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41978 (request call id 326) took 7113 ms. Trace:
W20250116 02:59:40.083760 23409 log.cc:926] Time spent T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.172s user 0.000s sys 0.000s
W20250116 02:59:40.060870 23313 consensus_meta.cc:287] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.613s user 0.015s sys 0.004s
W20250116 02:59:40.078428 23397 log.cc:926] Time spent T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.566s user 0.001s sys 0.000s
I20250116 02:59:40.085678 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:40.102130 23400 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.262s user 0.001s sys 0.001s
W20250116 02:59:40.079169 23406 log.cc:926] Time spent T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.231s user 0.000s sys 0.000s
W20250116 02:59:40.094542 23317 consensus_meta.cc:287] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.546s user 0.013s sys 0.002s
I20250116 02:59:39.955469 23426 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:40.077808 23405 log.cc:926] Time spent T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.525s user 0.000s sys 0.000s
W20250116 02:59:40.240752 20676 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.154s user 0.010s sys 0.003s
W20250116 02:59:40.375903 20676 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.289s user 0.010s sys 0.003s
W20250116 02:59:40.241112 21299 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.791s user 0.008s sys 0.004s
I20250116 02:59:40.241533 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.378718 20681 raft_consensus.cc:2363] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5.
I20250116 02:59:39.946105 23355 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:39.943913 23403 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:40.084971 22791 raft_consensus.cc:2798] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:40.087849 23428 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:40.113708 23413 log.cc:926] Time spent T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.156s user 0.002s sys 0.000s
W20250116 02:59:40.124204 23416 log.cc:926] Time spent T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.168s user 0.001s sys 0.002s
I20250116 02:59:39.730901 23275 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.245818 20749 raft_consensus.cc:2405] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20250116 02:59:40.243386 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.074419 23107 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.279415 22871 consensus_queue.cc:237] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:40.220080 22821 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.868s user 0.000s sys 0.013s
I20250116 02:59:40.098044 23256 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:40.103111 22753 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:40.167934 23257 consensus_meta.cc:287] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.002s user 0.008s sys 0.004s
I20250116 02:59:40.348577 23313 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.079537 23423 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:40.395807 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1976) took 1265 ms. Trace:
I20250116 02:59:40.394495 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.404431 23428 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:40.406129 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.409646 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.265482 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.404577 23403 raft_consensus.cc:695] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:40.061095 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:40.411187 23423 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.412369 23275 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.412000 23403 consensus_queue.cc:237] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.391088 23426 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.413568 23423 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.273088 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.379853 23434 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:40.414553 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 471) took 2063 ms. Trace:
I20250116 02:59:40.415189 23426 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.091655 23351 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:40.279629 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:39.848701 23205 consensus_queue.cc:237] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.415105 20682 rpcz_store.cc:276] 0116 02:59:38.351643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:38.351881 (+ 238us) service_pool.cc:225] Handling call
0116 02:59:38.352298 (+ 417us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:38.352708 (+ 410us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:39.132223 (+779515us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b1800300dc0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd3305 00007fc32a6e4107 00007fc326d198f7 00007fc326d2900f 00007fc326d28f88
0116 02:59:39.313489 (+181266us) log.cc:844] Serialized 14 byte log entry
0116 02:59:39.634923 (+321434us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:39.635675 (+ 752us) log.cc:844] Serialized 26 byte log entry
0116 02:59:39.635860 (+ 185us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:39.635893 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:39.635979 (+ 86us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:40.112712 (+476733us) spinlock_profiling.cc:243] Waited 3.68 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:40.414316 (+301604us) raft_consensus.cc:1680] finished
0116 02:59:40.414341 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:40.414487 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19180,"prepare.queue_time_us":792708,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":35219,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":2209,"spinlock_wait_cycles":22941952,"thread_start_us":427250,"threads_started":3,"wal-append.queue_time_us":640737}
I20250116 02:59:40.417434 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1661) took 1348 ms. Trace:
I20250116 02:59:40.409677 23363 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:40.410741 22821 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.059s user 0.001s sys 0.015s
I20250116 02:59:40.415518 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.312649 23248 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.419575 23351 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.420291 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.421236 23248 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.421895 23351 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.306977 23412 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.422745 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 2004) took 1075 ms. Trace:
I20250116 02:59:40.423286 21328 raft_consensus.cc:2363] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5.
I20250116 02:59:40.425364 23313 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:40.402349 23388 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.015s sys 0.019s
W20250116 02:59:40.426286 23388 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s user 0.015s sys 0.019s
W20250116 02:59:40.275518 20674 rpcz_store.cc:269] 0116 02:59:37.284316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.303864 (+ 19548us) service_pool.cc:225] Handling call
0116 02:59:40.085248 (+2781384us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001c5cd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:40.085388 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2495308544}
I20250116 02:59:40.426596 23429 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:40.406713 22519 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:40.410872 22791 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:40.403867 23107 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:40.427836 22791 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:40.398914 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.277256 23364 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:40.429632 20751 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:40.430173 20603 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:40.422384 20755 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:40.421759 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:40.431571 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:40.432433 20603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 444) took 2139 ms (client timeout 1742 ms). Trace:
I20250116 02:59:40.296485 20756 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:40.432770 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.433519 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.433848 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 2012) took 1084 ms. Trace:
W20250116 02:59:40.314350 20607 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 446) took 1948 ms (client timeout 1724 ms). Trace:
I20250116 02:59:40.318094 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.419096 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.435039 20599 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:40.435137 20750 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:40.417841 20676 rpcz_store.cc:276] 0116 02:59:39.068731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:39.073225 (+ 4494us) service_pool.cc:225] Handling call
0116 02:59:39.073656 (+ 431us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:39.073944 (+ 288us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:39.073971 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:39.074545 (+ 574us) log.cc:844] Serialized 26 byte log entry
0116 02:59:40.378556 (+1304011us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:40.378598 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.378701 (+ 103us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:40.417230 (+ 38529us) raft_consensus.cc:1680] finished
0116 02:59:40.417252 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:40.417379 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1289342,"threads_started":1,"wal-append.queue_time_us":1287811}
I20250116 02:59:40.277994 22957 raft_consensus.cc:738] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:40.348666 20600 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.006s sys 0.021s
I20250116 02:59:40.435976 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.402196 23273 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:40.356659 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.437173 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.437491 20752 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:40.321866 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:40.434784 20607 rpcz_store.cc:269] 0116 02:59:38.366350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.082802 (+1716452us) service_pool.cc:225] Handling call
0116 02:59:40.314240 (+231438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.284341 23245 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.436973 22957 consensus_queue.cc:260] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.360626 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:40.420255 23410 log.cc:926] Time spent T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250116 02:59:40.361712 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.266186 23311 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.434255 20755 rpcz_store.cc:276] 0116 02:59:39.349237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.432514 (+1083277us) service_pool.cc:225] Handling call
0116 02:59:40.433786 (+ 1272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.442567 20602 raft_consensus.cc:2388] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:40.442929 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.423142 20749 rpcz_store.cc:276] 0116 02:59:39.347178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:39.999093 (+651915us) service_pool.cc:225] Handling call
0116 02:59:40.422677 (+423584us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.358837 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:40.439653 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.444377 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.271209 20753 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:40.443033 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 459) took 2873 ms. Trace:
I20250116 02:59:40.403846 20757 rpcz_store.cc:276] 0116 02:59:39.130156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:39.130389 (+ 233us) service_pool.cc:225] Handling call
0116 02:59:39.130918 (+ 529us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:39.133122 (+ 2204us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:39.133208 (+ 86us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:39.134024 (+ 816us) log.cc:844] Serialized 26 byte log entry
0116 02:59:39.317135 (+183111us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:39.559103 (+241968us) log.cc:844] Serialized 14 byte log entry
0116 02:59:39.632552 (+ 73449us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:39.632632 (+ 80us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:40.395556 (+762924us) raft_consensus.cc:1680] finished
0116 02:59:40.395579 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:40.395739 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83099,"prepare.queue_time_us":626720,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":400993,"raft.run_cpu_time_us":420,"raft.run_wall_time_us":37929,"thread_start_us":467736,"threads_started":3,"wal-append.queue_time_us":353586}
I20250116 02:59:40.108726 23390 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.155880 20748 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:40.435252 23107 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:40.318924 23369 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.445124 20607 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:40.446656 23428 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:40.447260 23428 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:40.447556 20607 raft_consensus.cc:3049] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:40.265364 23419 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250116 02:59:40.436264 20600 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s user 0.006s sys 0.021s
I20250116 02:59:40.436894 23273 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.446223 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.448053 23424 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:40.445562 20680 rpcz_store.cc:276] 0116 02:59:37.569405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.569802 (+ 397us) service_pool.cc:225] Handling call
0116 02:59:37.570237 (+ 435us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:37.570650 (+ 413us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:37.570683 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:37.606427 (+ 35744us) log.cc:844] Serialized 52 byte log entry
0116 02:59:39.294917 (+1688490us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b1800352000. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32698be4b 00007fc326991913 00007fc326991cc1 00007fc326ce345f
0116 02:59:39.522053 (+227136us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:39.522100 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:39.522175 (+ 75us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:40.442700 (+920525us) raft_consensus.cc:1680] finished
0116 02:59:40.442726 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:40.442964 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":42382,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":22988,"spinlock_wait_cycles":6530432,"thread_start_us":226872,"threads_started":1,"wal-append.queue_time_us":2632840}
I20250116 02:59:40.450350 23273 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.447981 20607 raft_consensus.cc:738] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:40.446101 22519 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.451663 20607 consensus_queue.cc:260] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.452554 23412 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.452745 20607 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:40.410710 23335 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.427369 23429 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:40.380003 21299 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.941s user 0.008s sys 0.004s
I20250116 02:59:40.432659 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.295601 20754 rpcz_store.cc:276] 0116 02:59:39.184732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:39.184851 (+ 119us) service_pool.cc:225] Handling call
0116 02:59:40.130344 (+945493us) spinlock_profiling.cc:243] Waited 33.5 ms on lock 0x7b58001f0598. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:40.130675 (+ 331us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.294905 (+164230us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.295136 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76953984}
I20250116 02:59:40.459795 20601 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:40.460295 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.461035 20754 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:40.461146 20601 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250116 02:59:40.430212 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1998) took 1235 ms. Trace:
I20250116 02:59:40.461563 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1991) took 1273 ms. Trace:
I20250116 02:59:40.462329 20751 rpcz_store.cc:276] 0116 02:59:39.225482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:39.791023 (+565541us) service_pool.cc:225] Handling call
0116 02:59:40.398688 (+607665us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b58001f0598. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:40.430131 (+ 31443us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":449177344}
I20250116 02:59:40.463837 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.464534 20751 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:40.465481 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.466080 20751 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:40.462884 20754 rpcz_store.cc:276] 0116 02:59:39.188476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.459955 (+1271479us) service_pool.cc:225] Handling call
0116 02:59:40.461496 (+ 1541us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.466643 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1986) took 1280 ms. Trace:
I20250116 02:59:40.467458 20751 rpcz_store.cc:276] 0116 02:59:39.186704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.465187 (+1278483us) service_pool.cc:225] Handling call
0116 02:59:40.466544 (+ 1357us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.468545 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:40.439874 23434 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.440030 20608 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:40.444186 23311 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.434397 20756 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:59:40.271637 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.013s sys 0.009s
I20250116 02:59:40.372339 23317 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:40.446185 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 521) took 1167 ms. Trace:
I20250116 02:59:40.438314 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.390193 21333 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:40.485008 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 2011) took 1136 ms. Trace:
I20250116 02:59:40.440837 23245 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:40.454838 20757 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:40.445592 20749 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:40.489210 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:40.489996 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1992) took 1301 ms. Trace:
I20250116 02:59:40.466697 20646 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:40.490337 20749 rpcz_store.cc:276] 0116 02:59:39.188765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.488952 (+1300187us) service_pool.cc:225] Handling call
0116 02:59:40.489933 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.485402 20756 rpcz_store.cc:276] 0116 02:59:39.348990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.433274 (+1084284us) service_pool.cc:225] Handling call
0116 02:59:40.484936 (+ 51662us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.499583 20680 raft_consensus.cc:1235] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250116 02:59:40.438354 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.502430 20752 raft_consensus.cc:2405] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20250116 02:59:40.435667 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 2005) took 1088 ms. Trace:
I20250116 02:59:40.503014 20750 rpcz_store.cc:276] 0116 02:59:39.347492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.418808 (+1071316us) service_pool.cc:225] Handling call
0116 02:59:40.435605 (+ 16797us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.504559 20750 raft_consensus.cc:1270] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250116 02:59:40.505223 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1994) took 1315 ms. Trace:
I20250116 02:59:40.505533 20750 rpcz_store.cc:276] 0116 02:59:39.189278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.503810 (+1314532us) service_pool.cc:225] Handling call
0116 02:59:40.504152 (+ 342us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.504924 (+ 772us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.505147 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.089972 23338 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:40.445562 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 2016) took 1095 ms. Trace:
W20250116 02:59:40.456738 20603 rpcz_store.cc:269] 0116 02:59:38.293484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:38.293689 (+ 205us) service_pool.cc:225] Handling call
0116 02:59:40.432239 (+2138550us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b5400055fd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:40.432336 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4909792640}
I20250116 02:59:40.516943 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:40.440713 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.467103 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:40.516846 20753 rpcz_store.cc:276] 0116 02:59:39.383104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.022020 (+638916us) service_pool.cc:225] Handling call
0116 02:59:40.445499 (+423479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.517536 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:40.518209 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 2015) took 1168 ms. Trace:
I20250116 02:59:40.518586 20754 rpcz_store.cc:276] 0116 02:59:39.349989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.466844 (+1116855us) service_pool.cc:225] Handling call
0116 02:59:40.518141 (+ 51297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.519193 20750 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250116 02:59:40.520210 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 520) took 1292 ms. Trace:
I20250116 02:59:40.520620 20754 rpcz_store.cc:276] 0116 02:59:39.227957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.519262 (+1291305us) service_pool.cc:225] Handling call
0116 02:59:40.519579 (+ 317us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.519844 (+ 265us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:40.519872 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:40.519923 (+ 51us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:40.519973 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.520028 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:40.520144 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.462581 20601 raft_consensus.cc:1235] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6. Ops: []
W20250116 02:59:40.485570 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.019s user 0.015s sys 0.015s
I20250116 02:59:40.528759 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1995) took 1339 ms. Trace:
I20250116 02:59:40.529830 20750 rpcz_store.cc:276] 0116 02:59:39.189534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.506363 (+1316829us) service_pool.cc:225] Handling call
0116 02:59:40.514524 (+ 8161us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:40.528251 (+ 13727us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.528669 (+ 418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.532362 20683 raft_consensus.cc:1270] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250116 02:59:40.532666 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.385131 20604 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:40.507302 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:40.492105 20749 raft_consensus.cc:1270] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250116 02:59:40.440873 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.534874 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:40.535566 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1990) took 1347 ms. Trace:
I20250116 02:59:40.535748 20605 raft_consensus.cc:2405] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20250116 02:59:40.533203 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.444432 20602 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 02:59:40.536887 20749 rpcz_store.cc:276] 0116 02:59:39.187926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.491248 (+1303322us) service_pool.cc:225] Handling call
0116 02:59:40.491718 (+ 470us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.535296 (+ 43578us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.535508 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.536532 20757 raft_consensus.cc:2405] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250116 02:59:40.534991 20755 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:40.522430 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2002) took 1181 ms. Trace:
I20250116 02:59:40.529325 20677 raft_consensus.cc:2388] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:40.469435 20751 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:40.439774 22957 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:40.554576 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 519) took 1327 ms. Trace:
I20250116 02:59:40.555065 20753 rpcz_store.cc:276] 0116 02:59:39.227596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.518326 (+1290730us) service_pool.cc:225] Handling call
0116 02:59:40.550578 (+ 32252us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.554218 (+ 3640us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:40.554257 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:40.554302 (+ 45us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:40.554341 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.554395 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:40.554513 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.540169 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:40.557556 20599 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:40.448046 23369 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.563841 20608 raft_consensus.cc:1212] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20250116 02:59:40.553313 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.564540 20608 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250116 02:59:40.401366 20675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34864 (request call id 1178) took 2155 ms. Trace:
I20250116 02:59:40.527977 23419 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:40.550372 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1996) took 1360 ms. Trace:
I20250116 02:59:40.559077 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.554860 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.597893 20757 rpcz_store.cc:276] 0116 02:59:39.189780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.488225 (+1298445us) service_pool.cc:225] Handling call
0116 02:59:40.550275 (+ 62050us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.538794 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 1987) took 1351 ms. Trace:
I20250116 02:59:40.605017 20751 rpcz_store.cc:276] 0116 02:59:39.186963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.468240 (+1281277us) service_pool.cc:225] Handling call
0116 02:59:40.538698 (+ 70458us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.569089 20607 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250116 02:59:40.570323 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.611452 20677 raft_consensus.cc:2405] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250116 02:59:40.569785 20756 raft_consensus.cc:1270] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250116 02:59:40.571348 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.576905 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.613215 20683 raft_consensus.cc:2388] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:40.613554 20678 raft_consensus.cc:2405] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20250116 02:59:40.583294 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:40.584101 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 2010) took 1235 ms. Trace:
I20250116 02:59:40.584124 23434 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.585965 20502 catalog_manager.cc:5526] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 4 to 5, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 5 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:40.588730 23422 consensus_queue.cc:1035] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20250116 02:59:40.589188 20599 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:40.572665 20675 rpcz_store.cc:276] 0116 02:59:38.245962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:38.248051 (+ 2089us) service_pool.cc:225] Handling call
0116 02:59:40.400654 (+2152603us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b54001c4dd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c94d50 00007fc326d17f2a 00007fc32bc71c7e 00007fc32534e817 00007fc32534e7a2 00007fc32534e6d1 00007fc32534e64c 00007fc32534d902 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:40.401244 (+ 590us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4422924160}
I20250116 02:59:40.589538 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.595233 23335 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:40.595103 20750 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:40.595865 23256 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:40.309505 20504 rpcz_store.cc:276] 0116 02:59:33.025728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:33.105560 (+ 79832us) service_pool.cc:225] Handling call
0116 02:59:40.074453 (+6968893us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:35.622071 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:35.637571 (+ 15500us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:35.637634 (+ 63us) write_op.cc:435] Acquired schema lock
0116 02:59:35.637667 (+ 33us) tablet.cc:659] Decoding operations
0116 02:59:35.898266 (+260599us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:35.898368 (+ 102us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:35.898408 (+ 40us) tablet.cc:682] Acquiring locks for 18 operations
0116 02:59:35.900204 (+ 1796us) tablet.cc:698] Row locks acquired
0116 02:59:35.900245 (+ 41us) write_op.cc:260] PREPARE: finished
0116 02:59:35.900477 (+ 232us) write_op.cc:270] Start()
0116 02:59:35.900659 (+ 182us) write_op.cc:276] Timestamp: P: 1736996375900399 usec, L: 0
0116 02:59:35.900680 (+ 21us) op_driver.cc:348] REPLICATION: starting
0116 02:59:35.901666 (+ 986us) log.cc:844] Serialized 7133 byte log entry
0116 02:59:37.410861 (+1509195us) op_driver.cc:464] REPLICATION: finished
0116 02:59:38.060361 (+649500us) write_op.cc:301] APPLY: starting
0116 02:59:38.060483 (+ 122us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:38.154497 (+ 94014us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:38.162493 (+ 7996us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:38.302002 (+139509us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:38.302411 (+ 409us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0116 02:59:38.302461 (+ 50us) write_op.cc:312] APPLY: finished
0116 02:59:38.303053 (+ 592us) log.cc:844] Serialized 125 byte log entry
0116 02:59:38.396621 (+ 93568us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:38.396863 (+ 242us) write_op.cc:454] Released schema lock
0116 02:59:38.396965 (+ 102us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:38.453618 (+ 56653us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:38.457651 (+ 4033us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":34,"child_traces":[["op",{"apply.queue_time_us":604244,"apply.run_cpu_time_us":20960,"apply.run_wall_time_us":391861,"num_ops":18,"prepare.queue_time_us":187231,"prepare.run_cpu_time_us":24860,"prepare.run_wall_time_us":1128430,"raft.queue_time_us":273895,"raft.run_cpu_time_us":13241,"raft.run_wall_time_us":382964,"replication_time_us":1509761,"thread_start_us":1527295,"threads_started":5,"wal-append.queue_time_us":1273528,"wal-append.run_cpu_time_us":36546,"wal-append.run_wall_time_us":2531375}]]}
I20250116 02:59:40.538117 20754 rpcz_store.cc:276] 0116 02:59:39.340767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.521531 (+1180764us) service_pool.cc:225] Handling call
0116 02:59:40.521869 (+ 338us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.522090 (+ 221us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:40.522116 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:40.522162 (+ 46us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:40.522203 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.522255 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:40.522367 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.555914 23273 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:40.673987 20647 leader_election.cc:336] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.561s (SENT)
I20250116 02:59:40.447698 23428 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:40.699080 20674 raft_consensus.cc:1270] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250116 02:59:40.669952 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 1993) took 1480 ms. Trace:
I20250116 02:59:40.674217 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:40.488847 20748 rpcz_store.cc:276] 0116 02:59:39.279095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:39.485723 (+206628us) service_pool.cc:225] Handling call
0116 02:59:40.446116 (+960393us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:40.455838 23424 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:40.575294 22519 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:40.687913 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:40.613754 23357 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:40.533291 23338 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:40.773418 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 464605us
W20250116 02:59:40.859032 21302 leader_election.cc:336] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.562s (SENT)
I20250116 02:59:41.019409 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:41.021124 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 162167us
I20250116 02:59:41.021821 20679 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:41.022586 20756 rpcz_store.cc:276] 0116 02:59:39.189027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.496103 (+1307076us) service_pool.cc:225] Handling call
0116 02:59:40.566958 (+ 70855us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.612862 (+ 45904us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.669727 (+ 56865us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b5400396550. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:40.669885 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97599872}
I20250116 02:59:41.019795 20607 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:41.025368 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:41.023608 20502 catalog_manager.cc:5526] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 3 to 4, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 4 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:41.030745 20751 raft_consensus.cc:1270] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250116 02:59:40.962649 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:41.029601 20749 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250116 02:59:41.032102 20647 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:40.819566 21334 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:41.022708 20755 rpcz_store.cc:276] 0116 02:59:39.348742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.442681 (+1093939us) service_pool.cc:225] Handling call
0116 02:59:40.583830 (+141149us) spinlock_profiling.cc:243] Waited 91.3 ms on lock 0x7b540033d6d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:40.584021 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":210022272}
W20250116 02:59:40.774374 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 166059us
W20250116 02:59:40.909418 20720 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.003s sys 0.014s
W20250116 02:59:41.044030 20720 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.725s user 0.003s sys 0.018s
I20250116 02:59:41.057546 20602 raft_consensus.cc:1270] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20250116 02:59:41.068380 20720 connection.cc:232] RPC call timeout handler was delayed by -4.2688401300000578s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1544s for a response
W20250116 02:59:41.069087 20720 connection.cc:232] RPC call timeout handler was delayed by -4.1680401300000085s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1656s for a response
W20250116 02:59:41.069576 20720 connection.cc:232] RPC call timeout handler was delayed by -4.0780401299999767s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20250116 02:59:41.069944 20720 connection.cc:232] RPC call timeout handler was delayed by -4.0438401300000351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1794s for a response
W20250116 02:59:41.070302 20720 connection.cc:232] RPC call timeout handler was delayed by -3.8962401300000238s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1958s for a response
W20250116 02:59:41.070683 20720 connection.cc:232] RPC call timeout handler was delayed by -3.8539401300000691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2005s for a response
W20250116 02:59:41.071060 20720 connection.cc:232] RPC call timeout handler was delayed by -3.646040130000074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22360000000000002s for a response
W20250116 02:59:41.071421 20720 connection.cc:232] RPC call timeout handler was delayed by -3.5173401299999796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2379s for a response
I20250116 02:59:41.064308 20502 catalog_manager.cc:5526] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 2 to 6, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 6 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:59:41.072995 20502 catalog_manager.cc:5526] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 3 to 6, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 6 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:59:41.084496 20502 catalog_manager.cc:5526] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: config changed from index 3 to 5, term changed from 3 to 4, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to <none>, 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:41.087050 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:41.088459 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 527265us
I20250116 02:59:41.089314 21299 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:40.962445 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:41.064496 22998 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.001s sys 0.035s
W20250116 02:59:41.091125 22998 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.687s user 0.002s sys 0.035s
I20250116 02:59:41.091410 20502 catalog_manager.cc:5526] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 2, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 2 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:59:41.093340 20502 catalog_manager.cc:5526] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:41.093753 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:41.021860 21330 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:41.398188 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:41.403239 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 729207us
I20250116 02:59:41.115869 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:41.213604 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2003) took 1866 ms. Trace:
W20250116 02:59:41.417932 20647 leader_election.cc:336] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.594s (SENT)
I20250116 02:59:41.275166 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:41.115396 21335 raft_consensus.cc:1235] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:41.416482 20754 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 5
I20250116 02:59:41.397208 20646 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:41.102409 20502 catalog_manager.cc:5526] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:41.420504 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:41.419919 20603 raft_consensus.cc:2388] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:59:41.305783 20722 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.013s sys 0.010s
W20250116 02:59:41.474664 20722 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.086s user 0.023s sys 0.015s
W20250116 02:59:41.303480 23403 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.017s sys 0.033s
W20250116 02:59:41.071759 20720 connection.cc:232] RPC call timeout handler was delayed by -3.4120401300000367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24960000000000002s for a response
I20250116 02:59:41.141337 20750 raft_consensus.cc:1270] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250116 02:59:41.484558 20720 connection.cc:232] RPC call timeout handler was delayed by -3.318440129999999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26s for a response
W20250116 02:59:41.485805 20720 connection.cc:232] RPC call timeout handler was delayed by -2.9080401300000176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30560000000000004s for a response
W20250116 02:59:41.078943 22871 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.007s sys 0.014s
I20250116 02:59:41.412073 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2154) took 1044 ms. Trace:
W20250116 02:59:41.216146 20682 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.000s sys 0.016s
W20250116 02:59:41.489917 20682 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.951s user 0.001s sys 0.024s
I20250116 02:59:41.066012 23434 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:41.124150 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:41.097295 23417 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:41.095422 23426 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:41.418227 20752 rpcz_store.cc:276] 0116 02:59:39.346769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.528051 (+1181282us) service_pool.cc:225] Handling call
0116 02:59:40.572280 (+ 44229us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:41.173972 (+601692us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:41.174003 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:41.174043 (+ 40us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:41.174082 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:41.213172 (+ 39090us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b5400395b54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:41.213261 (+ 89us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:41.213497 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1383123840}
W20250116 02:59:41.489100 22871 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.063s user 0.011s sys 0.026s
I20250116 02:59:41.508843 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2021) took 1779 ms. Trace:
W20250116 02:59:40.983968 23437 log.cc:926] Time spent T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.294s user 0.001s sys 0.000s
I20250116 02:59:41.483783 23419 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:40.824013 23311 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:41.397148 20649 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:41.400715 22957 consensus_meta.cc:287] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.836s user 0.008s sys 0.003s
I20250116 02:59:41.475956 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:41.094527 22753 consensus_meta.cc:287] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.688s user 0.006s sys 0.002s
I20250116 02:59:41.488809 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:41.480396 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 50032.4us
W20250116 02:59:41.486158 20720 connection.cc:232] RPC call timeout handler was delayed by -2.5759401300000491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3425s for a response
W20250116 02:59:41.517758 20720 connection.cc:232] RPC call timeout handler was delayed by -2.1880401299999903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3856s for a response
W20250116 02:59:41.518203 20572 leader_election.cc:336] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.618s (SENT)
I20250116 02:59:41.464758 20647 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:41.518679 20572 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:41.518419 20720 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:41.216817 21327 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.012s sys 0.029s
W20250116 02:59:41.486505 20681 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.899s user 0.010s sys 0.024s
W20250116 02:59:41.397989 23205 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.942s user 0.020s sys 0.034s
W20250116 02:59:41.180289 23294 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.018s sys 0.007s
I20250116 02:59:41.423558 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:41.064744 23317 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:41.489195 20606 rpcz_store.cc:276] 0116 02:59:40.367605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.544277 (+176672us) service_pool.cc:225] Handling call
0116 02:59:41.411378 (+867101us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:41.411658 (+ 280us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:41.411687 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:41.411733 (+ 46us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:41.411775 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:41.411845 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:41.411997 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:41.419684 23414 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:41.103010 23429 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:41.466378 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:41.492559 20607 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:41.422485 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:41.484484 23403 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.066s user 0.019s sys 0.043s
I20250116 02:59:41.491464 23433 consensus_queue.cc:1046] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20250116 02:59:41.496517 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:41.279685 23441 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:41.077256 23273 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:41.498281 22993 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 02:59:41.287005 23439 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:41.066394 23335 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.580952 20604 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250116 02:59:41.066502 23357 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:41.066210 23256 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.499785 23434 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:41.472939 20502 catalog_manager.cc:5526] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 2 to 3, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 3 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250116 02:59:41.507989 20723 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.788s user 0.016s sys 0.006s
I20250116 02:59:41.508224 20754 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:41.103170 23107 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:41.509240 20750 rpcz_store.cc:276] 0116 02:59:39.729063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.531374 (+802311us) service_pool.cc:225] Handling call
0116 02:59:40.594968 (+ 63594us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:41.486732 (+891764us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:41.508620 (+ 21888us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b540033d6d0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:41.508762 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67066752}
W20250116 02:59:41.512146 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 115136us
I20250116 02:59:41.063897 23369 consensus_queue.cc:1035] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.427s
W20250116 02:59:41.464527 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 67436.3us
W20250116 02:59:41.522358 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 104486us
W20250116 02:59:41.603554 20647 leader_election.cc:336] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.656s (SENT)
W20250116 02:59:41.572741 23205 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.117s user 0.025s sys 0.040s
I20250116 02:59:41.606649 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:41.554519 20751 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250116 02:59:41.511390 23338 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:41.572026 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 53705.9us
I20250116 02:59:41.559360 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:41.615644 20720 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:41.615923 23434 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:41.616444 23434 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:41.616868 20720 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:41.617230 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:41.617952 20720 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:41.619263 23438 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:41.603969 20647 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:41.569829 20602 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 6
I20250116 02:59:41.617973 20674 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:41.573060 20608 raft_consensus.cc:2363] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:41.572503 23429 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:41.584548 23273 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:41.584911 23403 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:41.571672 23294 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.173s user 0.022s sys 0.017s
I20250116 02:59:41.587177 20676 raft_consensus.cc:1270] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250116 02:59:41.643821 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:41.503290 23381 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.060s
W20250116 02:59:41.598412 20723 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.878s user 0.016s sys 0.006s
I20250116 02:59:41.598433 20502 catalog_manager.cc:5526] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:59:41.601115 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2143) took 2052 ms. Trace:
I20250116 02:59:41.603951 23386 raft_consensus.cc:3049] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:41.515960 23436 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:41.550949 23426 raft_consensus.cc:2743] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:41.572893 21327 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.090s user 0.014s sys 0.038s
W20250116 02:59:41.515210 20605 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.807s user 0.007s sys 0.016s
I20250116 02:59:41.521808 23419 raft_consensus.cc:695] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:41.572795 20680 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250116 02:59:41.621078 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:41.568562 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 48801.7us
W20250116 02:59:41.573562 20681 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.987s user 0.010s sys 0.024s
I20250116 02:59:41.625085 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43854 (request call id 2161) took 1064 ms. Trace:
I20250116 02:59:41.505685 23417 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:41.580533 23414 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.644652 20606 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:41.677907 20602 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:41.514472 23428 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:41.601301 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1705) took 1506 ms. Trace:
I20250116 02:59:41.648808 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:41.646844 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 927154us
I20250116 02:59:41.595778 23424 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:41.647327 20600 rpcz_store.cc:276] 0116 02:59:39.549185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:39.549409 (+ 224us) service_pool.cc:225] Handling call
0116 02:59:39.549854 (+ 445us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:39.787743 (+237889us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:39.787777 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:39.804700 (+ 16923us) log.cc:844] Serialized 26 byte log entry
0116 02:59:40.104938 (+300238us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b48000fbf58. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32bbd35a1 00007fc32bbca687 00007fc32bbca51a 00007fc32bcbe9df 00007fc32bcbe6a7 00007fc32bcd2bac 00007fc32bcd2b58 00007fc32bcd2ad8 00007fc32bcd2a93 00007fc32bcd1d3c 00007fc32ac3ed16
0116 02:59:40.448876 (+343938us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:40.448943 (+ 67us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:40.449013 (+ 70us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:41.588868 (+1139855us) raft_consensus.cc:1680] finished
0116 02:59:41.588898 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:41.601034 (+ 12136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":180455,"raft.queue_time_us":298904,"raft.run_cpu_time_us":2344,"raft.run_wall_time_us":240158,"spinlock_wait_cycles":26812288,"thread_start_us":643761,"threads_started":1,"wal-append.queue_time_us":755369}
I20250116 02:59:41.513407 23311 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.695654 23419 consensus_queue.cc:237] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:41.696447 20723 connection.cc:232] RPC call timeout handler was delayed by -1.123834611999996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19910000000000003s for a response
W20250116 02:59:41.697229 20723 connection.cc:232] RPC call timeout handler was delayed by -1.0526059830000349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
I20250116 02:59:41.697901 23428 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:41.596407 23439 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.698745 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2157) took 1309 ms. Trace:
I20250116 02:59:41.698519 23428 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.699189 20601 rpcz_store.cc:276] 0116 02:59:40.389143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:41.019082 (+629939us) service_pool.cc:225] Handling call
0116 02:59:41.019423 (+ 341us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:41.697510 (+678087us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:41.697545 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:41.697587 (+ 42us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:41.697620 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:41.698466 (+ 846us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b540014e1d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:41.698513 (+ 47us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:41.698677 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1558923520}
I20250116 02:59:41.700650 23428 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:41.700757 23439 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:41.696960 23311 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:41.701485 23311 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:41.686807 23302 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.016s
I20250116 02:59:41.692143 20604 rpcz_store.cc:276] 0116 02:59:40.095102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.533962 (+438860us) service_pool.cc:225] Handling call
0116 02:59:40.534323 (+ 361us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:41.589504 (+1055181us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:41.601012 (+ 11508us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400109354. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:41.601237 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2406327680}
I20250116 02:59:41.702272 20723 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:41.704180 23439 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:41.605372 23107 raft_consensus.cc:2743] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:41.695223 23424 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:41.705158 23424 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:41.705674 23424 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:41.705112 23439 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:41.706310 23439 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:41.683326 23417 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:41.710728 20723 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:41.711709 23107 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:41.712116 23107 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:41.712464 23107 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:41.677599 23436 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:41.638504 23429 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:41.714905 23436 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:41.716027 23357 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:41.676069 20572 connection.cc:232] RPC call timeout handler was delayed by -3.776418585999977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20870000000000002s for a response
W20250116 02:59:41.718391 20572 leader_election.cc:336] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.090s (SENT)
I20250116 02:59:41.718931 20572 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 02:59:41.720162 20572 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.140s (SENT)
W20250116 02:59:41.721073 20572 connection.cc:232] RPC call timeout handler was delayed by -3.6605016459999433s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2595s for a response
W20250116 02:59:41.721426 20572 connection.cc:232] RPC call timeout handler was delayed by -3.6252708829999847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.298s for a response
W20250116 02:59:41.722152 20572 connection.cc:232] RPC call timeout handler was delayed by -3.346687793000001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
I20250116 02:59:41.676930 20608 rpcz_store.cc:276] 0116 02:59:40.561355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.565275 (+ 3920us) service_pool.cc:225] Handling call
0116 02:59:41.625003 (+1059728us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:41.637635 20720 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:41.639065 23338 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:41.686702 20649 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:41.648376 23434 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:41.686213 23256 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:41.648540 20502 catalog_manager.cc:5526] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 3 to 4, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250116 02:59:41.674400 20605 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.966s user 0.009s sys 0.016s
W20250116 02:59:41.722508 20572 connection.cc:232] RPC call timeout handler was delayed by -3.2908877930000244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
W20250116 02:59:41.742079 20572 connection.cc:232] RPC call timeout handler was delayed by -3.2323877930000435s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1761s for a response
W20250116 02:59:41.742483 20572 connection.cc:232] RPC call timeout handler was delayed by -3.1522877930000277s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18500000000000003s for a response
I20250116 02:59:41.589231 23441 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.747901 23434 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:41.651755 23386 raft_consensus.cc:738] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:41.729771 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:41.743275 20572 connection.cc:232] RPC call timeout handler was delayed by -2.3785990510000374s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1758s for a response
I20250116 02:59:41.743527 20649 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:41.743572 20502 catalog_manager.cc:5526] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 2 to 5, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 5 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250116 02:59:41.752562 20572 connection.cc:232] RPC call timeout handler was delayed by -1.9609990509999307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
W20250116 02:59:41.753096 20572 connection.cc:232] RPC call timeout handler was delayed by -1.252699051000036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3009s for a response
W20250116 02:59:41.753425 20572 connection.cc:232] RPC call timeout handler was delayed by -1.1239990509999416s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31520000000000004s for a response
W20250116 02:59:41.753695 20572 connection.cc:232] RPC call timeout handler was delayed by -1.0006990509999696s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3289s for a response
I20250116 02:59:41.753891 20502 catalog_manager.cc:5526] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250116 02:59:41.744971 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 107413us
I20250116 02:59:41.756513 20720 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:41.686476 22791 consensus_meta.cc:287] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.258s user 0.007s sys 0.007s
I20250116 02:59:41.649206 23403 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.755892 23434 consensus_queue.cc:1046] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.011s
I20250116 02:59:41.766918 23428 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:41.768712 21329 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:41.770843 23338 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:41.717104 23433 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20250116 02:59:41.773595 23335 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:41.775298 23414 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:41.776286 23386 consensus_queue.cc:260] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.731081 23311 raft_consensus.cc:2743] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:41.783269 23428 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:41.760413 20502 catalog_manager.cc:5526] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 reported cstate change: term changed from 2 to 4, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:59:41.782312 20649 leader_election.cc:400] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Vote denied by peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195) with higher term. Message: Invalid argument: T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4.
I20250116 02:59:41.772007 23429 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:41.751597 23256 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:41.784648 23398 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:41.784821 21302 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.762s user 0.003s sys 0.055s
I20250116 02:59:41.762257 20720 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:41.785456 23369 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:41.789155 23434 raft_consensus.cc:3049] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:41.791136 23338 raft_consensus.cc:695] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:41.804980 20720 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:41.805790 21302 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.783s user 0.003s sys 0.055s
I20250116 02:59:41.806394 23429 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:41.770872 23441 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:41.795531 23428 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:41.814458 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 792147us
I20250116 02:59:41.781190 22791 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:41.814865 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.753s user 0.020s sys 0.069s
I20250116 02:59:41.819221 20720 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:41.819084 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.797s user 0.020s sys 0.069s
W20250116 02:59:41.821913 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 800455us
I20250116 02:59:41.823175 20720 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:41.829752 20571 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:41.851462 23403 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:41.852210 23403 consensus_queue.cc:1035] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250116 02:59:41.854071 23381 consensus_queue.cc:1046] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250116 02:59:41.812605 23386 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:41.858090 23338 consensus_queue.cc:237] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:41.811069 23434 raft_consensus.cc:738] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:41.861887 23434 consensus_queue.cc:260] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:41.863350 23434 raft_consensus.cc:3054] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:41.813840 23369 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:41.808153 23398 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:41.813983 23311 consensus_queue.cc:1035] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.062s
I20250116 02:59:41.875327 23311 raft_consensus.cc:2798] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:41.876358 23311 raft_consensus.cc:695] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:41.877635 23311 consensus_queue.cc:237] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:41.808396 23256 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:41.879397 23205 consensus_queue.cc:1035] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 02:59:41.881013 23313 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250116 02:59:41.882849 21329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56674 (request call id 481) took 2091 ms (client timeout 1631 ms). Trace:
W20250116 02:59:41.883333 21329 rpcz_store.cc:269] 0116 02:59:39.791559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:39.791709 (+ 150us) service_pool.cc:225] Handling call
0116 02:59:41.882583 (+2090874us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54004df3d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:41.882760 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2957261568}
I20250116 02:59:41.856849 21302 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:41.903328 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 80277.6us
I20250116 02:59:41.902009 23429 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:41.906138 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:41.907624 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:41.908342 20603 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:41.901913 23444 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:59:41.894541 22791 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:41.905225 20720 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:41.829002 23441 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250116 02:59:41.900475 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 43688.7us
I20250116 02:59:41.916432 21302 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:41.917979 21302 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:41.869961 23369 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:41.880896 23256 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:41.929744 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:59:41.904570 23302 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:41.936029 20720 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:41.933197 23444 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:41.926931 22753 raft_consensus.cc:2743] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:42.000320 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 82414us
W20250116 02:59:41.980239 21331 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.883s user 0.012s sys 0.018s
I20250116 02:59:41.905475 23424 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:41.913372 23368 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.317s user 0.027s sys 0.041s
I20250116 02:59:41.914640 23417 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.019137 21302 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:42.028381 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 92476us
I20250116 02:59:42.008811 23205 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:42.065438 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 46406.3us
W20250116 02:59:42.066887 21333 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.049s user 0.021s sys 0.036s
W20250116 02:59:42.062546 23368 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.464s user 0.029s sys 0.047s
I20250116 02:59:41.942065 20603 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:42.069880 20720 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:42.140848 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 71092.3us
W20250116 02:59:42.025686 21331 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.929s user 0.013s sys 0.018s
I20250116 02:59:41.940941 23436 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.060969 23443 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:42.063454 22998 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.972s user 0.018s sys 0.014s
I20250116 02:59:42.011673 23439 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:42.138502 21333 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.120s user 0.021s sys 0.036s
I20250116 02:59:42.145678 20649 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:42.172080 20720 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:42.177246 20720 leader_election.cc:400] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5.
I20250116 02:59:42.176968 23245 consensus_queue.cc:1035] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 02:59:42.177734 20720 leader_election.cc:403] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250116 02:59:42.178988 20679 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250116 02:59:42.172866 22998 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.081s user 0.018s sys 0.015s
I20250116 02:59:42.179610 20720 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:42.180413 23447 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20250116 02:59:42.180800 23441 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5.
I20250116 02:59:42.147311 23444 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:42.201918 20677 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.003s sys 0.027s
W20250116 02:59:42.204064 20677 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.584s user 0.003s sys 0.027s
W20250116 02:59:42.217306 20682 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.009s sys 0.013s
I20250116 02:59:42.182138 23443 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:42.218189 20682 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s user 0.010s sys 0.014s
I20250116 02:59:42.228036 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1670) took 1735 ms. Trace:
W20250116 02:59:42.233314 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.009s sys 0.014s
W20250116 02:59:42.234163 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.738s user 0.010s sys 0.014s
W20250116 02:59:42.234870 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 744994us
I20250116 02:59:42.186494 23426 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:42.149828 23313 consensus_queue.cc:1035] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
W20250116 02:59:42.211583 23430 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.005s sys 0.028s
I20250116 02:59:42.201974 23429 raft_consensus.cc:2743] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:42.209542 23399 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.009s sys 0.014s
I20250116 02:59:42.194880 22791 raft_consensus.cc:2798] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:59:42.179448 23205 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
W20250116 02:59:42.261813 20646 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.015s sys 0.011s
W20250116 02:59:42.269541 20646 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.643s user 0.015s sys 0.012s
I20250116 02:59:42.270769 20646 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:42.232484 23335 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:42.254756 23430 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.641s user 0.007s sys 0.028s
W20250116 02:59:42.213056 21302 leader_election.cc:336] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.534s (SENT)
W20250116 02:59:42.326905 20681 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.003s sys 0.041s
W20250116 02:59:42.328914 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 116134us
W20250116 02:59:42.267248 23442 log.cc:926] Time spent T 221670297d85466bac4e5a9bfead9b18 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.259s user 0.001s sys 0.000s
W20250116 02:59:42.329528 20681 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s user 0.003s sys 0.042s
W20250116 02:59:42.331130 20749 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.851s user 0.030s sys 0.021s
W20250116 02:59:42.267807 23399 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.715s user 0.012s sys 0.014s
I20250116 02:59:42.221722 23441 consensus_queue.cc:1035] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.054s
W20250116 02:59:42.332897 20749 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.852s user 0.031s sys 0.021s
W20250116 02:59:42.272832 20599 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.005s sys 0.017s
W20250116 02:59:42.330647 23107 consensus_meta.cc:287] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.618s user 0.014s sys 0.000s
W20250116 02:59:42.275625 21327 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.012s sys 0.022s
W20250116 02:59:42.333801 20647 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.000s sys 0.024s
W20250116 02:59:42.505270 20647 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.881s user 0.000s sys 0.025s
W20250116 02:59:42.508788 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 905164us
W20250116 02:59:42.482950 20601 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.004s sys 0.011s
W20250116 02:59:42.511132 20601 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.710s user 0.005s sys 0.012s
W20250116 02:59:42.483495 20573 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.018s sys 0.031s
W20250116 02:59:42.476612 20502 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.530s user 0.002s sys 0.007s
W20250116 02:59:42.337877 20599 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s user 0.005s sys 0.017s
W20250116 02:59:42.481000 20572 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.010s sys 0.044s
W20250116 02:59:42.515774 20572 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.011s sys 0.044s
W20250116 02:59:42.516371 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 759340us
W20250116 02:59:42.479935 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.005s sys 0.016s
W20250116 02:59:42.519438 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.679s user 0.005s sys 0.016s
W20250116 02:59:42.520100 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 679531us
I20250116 02:59:42.484376 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:42.339882 20676 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.022s sys 0.024s
W20250116 02:59:42.523563 20676 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.843s user 0.022s sys 0.025s
I20250116 02:59:42.523370 20603 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:42.348148 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1668) took 1857 ms. Trace:
W20250116 02:59:42.335928 20605 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.018s sys 0.012s
W20250116 02:59:42.485761 20608 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.010s sys 0.010s
W20250116 02:59:42.503525 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 173896us
W20250116 02:59:42.527499 20608 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.636s user 0.010s sys 0.011s
I20250116 02:59:42.525887 20681 rpcz_store.cc:276] 0116 02:59:40.490256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.491491 (+ 1235us) service_pool.cc:225] Handling call
0116 02:59:40.585802 (+ 94311us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:40.586316 (+ 514us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.586570 (+ 254us) raft_consensus.cc:1529] Early marking committed up to index 4
0116 02:59:41.684314 (+1097744us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.347630 (+663316us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:42.347702 (+ 72us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:42.347772 (+ 70us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:42.347873 (+ 101us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:42.348068 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14959,"prepare.queue_time_us":1338697,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":1955,"raft.queue_time_us":15104,"spinlock_wait_cycles":49906816,"thread_start_us":1736052,"threads_started":2,"wal-append.queue_time_us":737671}
W20250116 02:59:42.514983 20573 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.036s user 0.019s sys 0.031s
W20250116 02:59:42.527102 20605 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s user 0.019s sys 0.012s
I20250116 02:59:42.228812 20682 rpcz_store.cc:276] 0116 02:59:40.492160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.492289 (+ 129us) service_pool.cc:225] Handling call
0116 02:59:40.492593 (+ 304us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.492811 (+ 218us) raft_consensus.cc:1529] Early marking committed up to index 2
0116 02:59:41.490366 (+997555us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.227457 (+737091us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:42.227613 (+ 156us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:42.227677 (+ 64us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:42.227773 (+ 96us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:42.227956 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6117,"prepare.queue_time_us":1428890,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":147,"spinlock_wait_cycles":8418560,"thread_start_us":1687804,"threads_started":2}
I20250116 02:59:42.545357 20599 raft_consensus.cc:1235] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
W20250116 02:59:42.341090 21299 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.024s sys 0.044s
W20250116 02:59:42.346719 21327 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.638s user 0.012s sys 0.022s
I20250116 02:59:42.566797 20571 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:42.331570 23256 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:42.256211 23426 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.639142 23462 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:42.640614 23462 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.641810 23311 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250116 02:59:42.562793 21299 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.881s user 0.025s sys 0.045s
W20250116 02:59:42.643224 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 961614us
W20250116 02:59:42.644112 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 92427.8us
I20250116 02:59:42.592434 23368 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:42.361876 23424 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:42.434188 23302 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:42.380654 23338 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.012s sys 0.014s
I20250116 02:59:42.436599 23398 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:42.433768 23386 consensus_meta.cc:287] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.575s user 0.008s sys 0.006s
W20250116 02:59:42.403563 23273 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.015s sys 0.005s
I20250116 02:59:42.345253 23107 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:42.390678 23436 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:42.436466 23434 consensus_meta.cc:287] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.573s user 0.012s sys 0.000s
W20250116 02:59:42.379781 23419 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.023s sys 0.046s
W20250116 02:59:42.674028 23419 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s user 0.024s sys 0.047s
W20250116 02:59:42.431017 23457 log.cc:926] Time spent T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20250116 02:59:42.589610 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 487) took 2231 ms. Trace:
I20250116 02:59:42.677609 23462 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:42.675657 21327 rpcz_store.cc:276] 0116 02:59:40.357526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.365342 (+ 7816us) service_pool.cc:225] Handling call
0116 02:59:40.416742 (+ 51400us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:40.417264 (+ 522us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:40.417302 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:40.417980 (+ 678us) log.cc:844] Serialized 52 byte log entry
0116 02:59:41.692886 (+1274906us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:42.563985 (+871099us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.587841 (+ 23856us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.588426 (+ 585us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:42.588731 (+ 305us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:42.588843 (+ 112us) raft_consensus.cc:1680] finished
0116 02:59:42.588859 (+ 16us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:42.589215 (+ 356us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":549260,"prepare.queue_time_us":764475,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":117,"raft.queue_time_us":37515,"raft.run_cpu_time_us":559,"raft.run_wall_time_us":41584,"spinlock_wait_cycles":6400,"thread_start_us":2072561,"threads_started":2,"wal-append.queue_time_us":1402844}
W20250116 02:59:42.430478 23455 log.cc:926] Time spent T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20250116 02:59:42.647540 23368 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:42.366459 23337 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.002s sys 0.016s
I20250116 02:59:42.642433 23311 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:42.626153 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 81515.7us
W20250116 02:59:42.433171 23459 log.cc:926] Time spent T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20250116 02:59:42.662094 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 144279us
W20250116 02:59:42.434715 23428 consensus_meta.cc:287] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.610s user 0.012s sys 0.001s
I20250116 02:59:42.695142 23428 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.700557 23428 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:42.702205 23429 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:42.703289 23429 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.711580 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:42.712700 23368 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:42.714824 23337 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.924s user 0.002s sys 0.016s
W20250116 02:59:42.715646 23273 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.910s user 0.016s sys 0.006s
I20250116 02:59:42.341106 23417 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:42.692824 23311 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:42.418852 23439 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:42.731844 20603 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:42.351227 22791 raft_consensus.cc:695] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:42.740643 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 494612us
W20250116 02:59:42.743569 23499 log.cc:926] Time spent T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20250116 02:59:42.743350 20720 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:42.743072 22791 consensus_queue.cc:237] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:42.750479 23338 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.799s user 0.013s sys 0.014s
I20250116 02:59:42.758165 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 497) took 1136 ms. Trace:
I20250116 02:59:42.758740 20676 rpcz_store.cc:276] 0116 02:59:41.621862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:41.644528 (+ 22666us) service_pool.cc:225] Handling call
0116 02:59:41.679531 (+ 35003us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:41.679792 (+ 261us) raft_consensus.cc:1529] Early marking committed up to index 7
0116 02:59:42.524340 (+844548us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.733152 (+208812us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:42.757505 (+ 24353us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:42.757638 (+ 133us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:42.757748 (+ 110us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:42.758065 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95496,"prepare.queue_time_us":262611,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":104888,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":24023,"thread_start_us":944557,"threads_started":2,"wal-append.queue_time_us":101268}
I20250116 02:59:42.761422 20572 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:42.769263 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:42.771000 23436 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:42.746177 21329 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:42.761687 23357 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250116 02:59:42.753809 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 460725us
I20250116 02:59:42.778713 23436 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.796370 23426 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:42.780033 23441 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:42.338800 23335 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:42.782130 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 247633us
W20250116 02:59:42.777756 20647 connection.cc:232] RPC call timeout handler was delayed by -1.2442158029999746s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
W20250116 02:59:42.816838 20647 connection.cc:232] RPC call timeout handler was delayed by -1.2051949760000298s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
I20250116 02:59:42.822134 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 874) took 2333 ms. Trace:
I20250116 02:59:42.823007 23403 consensus_queue.cc:1035] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20250116 02:59:42.822932 21333 rpcz_store.cc:276] 0116 02:59:40.488282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.500913 (+ 12631us) service_pool.cc:225] Handling call
0116 02:59:40.501286 (+ 373us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:41.017602 (+516316us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:41.017634 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:41.018119 (+ 485us) log.cc:844] Serialized 26 byte log entry
0116 02:59:42.175885 (+1157766us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:42.176032 (+ 147us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:42.176142 (+ 110us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:42.821537 (+645395us) raft_consensus.cc:1680] finished
0116 02:59:42.821583 (+ 46us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:42.821872 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58728,"raft.queue_time_us":561680,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":26594,"thread_start_us":1157430,"threads_started":1,"wal-append.queue_time_us":1328861}
I20250116 02:59:42.780685 23429 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:42.803678 23424 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:42.825724 23424 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.827535 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:42.829188 23424 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:42.799599 23337 raft_consensus.cc:3049] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:42.798497 23436 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:42.796619 23107 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:42.826428 23335 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:42.864236 20720 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:42.843410 23337 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:42.845979 23419 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:42.880270 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:42.871357 20647 leader_election.cc:336] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.528s (SENT)
I20250116 02:59:42.882169 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 530) took 1621 ms. Trace:
I20250116 02:59:42.887125 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1707) took 2785 ms. Trace:
I20250116 02:59:42.892239 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 495) took 1263 ms. Trace:
I20250116 02:59:42.892746 20677 rpcz_store.cc:276] 0116 02:59:41.618242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:41.618835 (+ 593us) service_pool.cc:225] Handling call
0116 02:59:41.619186 (+ 351us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:41.619524 (+ 338us) raft_consensus.cc:1529] Early marking committed up to index 7
0116 02:59:42.204558 (+585034us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.841085 (+636527us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:42.856246 (+ 15161us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:42.863779 (+ 7533us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:42.874037 (+ 10258us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:42.881582 (+ 7545us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":91565,"prepare.queue_time_us":320742,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":98509,"raft.run_cpu_time_us":497,"raft.run_wall_time_us":26716,"thread_start_us":1061043,"threads_started":2,"wal-append.queue_time_us":414122}
W20250116 02:59:42.896818 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 53819.5us
W20250116 02:59:42.898064 20647 leader_election.cc:336] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.584s (SENT)
I20250116 02:59:42.889299 20605 rpcz_store.cc:276] 0116 02:59:40.095718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.536384 (+440666us) service_pool.cc:225] Handling call
0116 02:59:40.577746 (+ 41362us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:40.577988 (+ 242us) raft_consensus.cc:1529] Early marking committed up to index 7
0116 02:59:41.741517 (+1163529us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.766660 (+1025143us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.818229 (+ 51569us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:42.830445 (+ 12216us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:42.863763 (+ 33318us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:42.873605 (+ 9842us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:42.881393 (+ 7788us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46131,"prepare.queue_time_us":1467672,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":115,"raft.queue_time_us":156698,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":12684,"thread_start_us":1902876,"threads_started":3,"wal-append.queue_time_us":681307}
I20250116 02:59:42.883831 20749 rpcz_store.cc:276] 0116 02:59:41.188980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:41.189253 (+ 273us) service_pool.cc:225] Handling call
0116 02:59:41.422788 (+233535us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:41.423065 (+ 277us) raft_consensus.cc:1529] Early marking committed up to index 7
0116 02:59:42.337432 (+914367us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.807079 (+469647us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.807469 (+ 390us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:42.807558 (+ 89us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:42.807660 (+ 102us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:42.807829 (+ 169us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:42.810055 (+ 2226us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1071214,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":100,"raft.queue_time_us":4643,"raft.run_cpu_time_us":968,"raft.run_wall_time_us":92272,"thread_start_us":1198708,"threads_started":2,"wal-append.queue_time_us":283137}
W20250116 02:59:42.903182 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 141996us
I20250116 02:59:42.911942 20647 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:42.892526 21331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 873) took 2408 ms. Trace:
I20250116 02:59:42.888149 23479 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:42.888260 22519 consensus_queue.cc:1046] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.044s
I20250116 02:59:42.900688 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:42.936041 22519 raft_consensus.cc:3049] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:42.936627 22519 raft_consensus.cc:738] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:42.901571 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:42.892589 22753 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:42.923682 23338 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20250116 02:59:42.924309 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:42.904009 23357 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:42.889966 23424 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:42.945960 20649 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:42.933017 23479 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:42.898521 23417 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.044s
I20250116 02:59:42.888329 23403 consensus_queue.cc:1035] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20250116 02:59:42.946249 23357 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:42.902802 23419 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.951864 23479 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
W20250116 02:59:42.960410 21302 leader_election.cc:336] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 2.011s (SENT)
I20250116 02:59:42.969645 21333 raft_consensus.cc:1235] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:42.970432 23337 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:42.959720 23273 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:42.982600 21327 raft_consensus.cc:2405] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250116 02:59:42.999831 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:43.000974 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 190579us
I20250116 02:59:43.002960 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:43.003629 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:43.004503 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:43.005381 20678 raft_consensus.cc:2363] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:43.006175 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:43.009593 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:43.011379 22993 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:43.012356 22993 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:43.018204 20749 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250116 02:59:43.001695 23509 log.cc:926] Time spent T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20250116 02:59:43.011581 20720 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:43.024969 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.025982 20754 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:43.027168 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:43.029503 23107 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:43.030225 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:43.031185 21329 raft_consensus.cc:2388] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 4: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:43.011587 23398 consensus_queue.cc:1046] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20250116 02:59:43.033285 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:43.033342 20720 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:43.020939 22993 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:43.024569 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:42.931294 21331 rpcz_store.cc:276] 0116 02:59:40.484079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:40.490136 (+ 6057us) service_pool.cc:225] Handling call
0116 02:59:40.669647 (+179511us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:41.095899 (+426252us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:41.095943 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:41.096799 (+ 856us) log.cc:844] Serialized 26 byte log entry
0116 02:59:41.101778 (+ 4979us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b48000b0c58. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32bbd35a1 00007fc32bbca687 00007fc32bbca51a 00007fc32bcbe9df 00007fc32bcbe6a7 00007fc32bcd2bac 00007fc32bcd2b58 00007fc32bcd2ad8 00007fc32bcd2a93 00007fc32bcd1d3c 00007fc32ac3ed16
0116 02:59:42.171152 (+1069374us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:42.851379 (+680227us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.891663 (+ 40284us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:42.891814 (+ 151us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:42.891966 (+ 152us) raft_consensus.cc:1680] finished
0116 02:59:42.892115 (+ 149us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:42.892332 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":431855,"prepare.queue_time_us":252032,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":430463,"raft.run_cpu_time_us":2034,"raft.run_wall_time_us":13817,"spinlock_wait_cycles":8278784,"thread_start_us":1555656,"threads_started":2,"wal-append.queue_time_us":1244209}
I20250116 02:59:42.985234 22791 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:43.033638 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:43.043145 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:43.052795 23107 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:43.053993 22871 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:43.054850 22871 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:43.050390 21329 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:43.056744 21335 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:43.057416 22871 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:43.058084 23506 log.cc:926] Time spent T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20250116 02:59:43.049026 20750 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:43.060397 20748 raft_consensus.cc:1235] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:43.063524 20601 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:43.050711 23337 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:43.069480 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:43.072018 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 174048us
W20250116 02:59:43.073182 20647 leader_election.cc:336] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.635s (SENT)
I20250116 02:59:43.073912 20647 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:43.075872 20647 leader_election.cc:336] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.713s (SENT)
I20250116 02:59:42.993499 22519 consensus_queue.cc:260] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:43.080091 22519 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:43.080816 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2174) took 1284 ms. Trace:
W20250116 02:59:43.077484 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 140255us
W20250116 02:59:43.078249 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 204868us
W20250116 02:59:43.086416 21302 leader_election.cc:336] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 2.228s (SENT)
I20250116 02:59:43.077409 23256 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:43.089056 20681 raft_consensus.cc:1235] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:43.077586 23338 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:43.094425 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:43.095726 20605 raft_consensus.cc:2405] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250116 02:59:43.097133 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:43.103266 23483 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:43.104125 20571 connection.cc:232] RPC call timeout handler was delayed by -1.2797356670000681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20250116 02:59:43.104763 20571 connection.cc:232] RPC call timeout handler was delayed by -1.2293356669999866s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
I20250116 02:59:43.104409 23483 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:43.106045 20571 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:43.103336 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:43.106853 23483 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:43.107412 20683 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:43.067281 22791 raft_consensus.cc:695] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:43.118845 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 95984.7us
W20250116 02:59:43.119024 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 73943.8us
I20250116 02:59:43.081432 20608 rpcz_store.cc:276] 0116 02:59:41.796374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:41.796870 (+ 496us) service_pool.cc:225] Handling call
0116 02:59:41.797308 (+ 438us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:41.891511 (+ 94203us) raft_consensus.cc:1529] Early marking committed up to index 3
0116 02:59:42.530463 (+638952us) log.cc:844] Serialized 14 byte log entry
0116 02:59:42.818600 (+288137us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:42.833884 (+ 15284us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:42.859997 (+ 26113us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:43.070140 (+210143us) spinlock_profiling.cc:243] Waited 9.14 ms on lock 0x7b18001c26a0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323da3ec6 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9
0116 02:59:43.073484 (+ 3344us) spinlock_profiling.cc:243] Waited 4.2 ms on lock 0x7b18001c26a0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd67e8 00007fc323dd9cc1 00007fc323dd9c79 00007fc323dd9c09 00007fc323dd9bd1 00007fc323dd8ecd 00007fc32ce13234 00007fc32ce13069 00007fc323db8fe6
0116 02:59:43.080446 (+ 6962us) spinlock_profiling.cc:243] Waited 93.9 ms on lock 0x7b54001f18d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:43.080508 (+ 62us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:43.080733 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33588,"prepare.queue_time_us":233734,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":40864,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":50200,"spinlock_wait_cycles":246592256,"thread_start_us":840171,"threads_started":2,"wal-append.queue_time_us":332686}
I20250116 02:59:43.118336 23398 raft_consensus.cc:3049] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
W20250116 02:59:43.136678 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 170306us
I20250116 02:59:43.143822 22791 consensus_queue.cc:237] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:43.146690 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:43.149724 20605 raft_consensus.cc:2405] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250116 02:59:43.138686 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:43.139444 23398 raft_consensus.cc:738] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:43.166277 23430 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 02:59:43.173516 23419 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:43.175491 21300 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:43.177577 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:43.185330 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 152410us
W20250116 02:59:43.194782 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 88853.3us
I20250116 02:59:43.197791 20572 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:43.202268 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 126424us
I20250116 02:59:43.200148 23430 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:43.204283 21329 raft_consensus.cc:2405] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20250116 02:59:43.221789 20647 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.853s (SENT)
I20250116 02:59:43.220463 21300 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:43.229736 20647 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:43.220402 23430 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:43.234647 20649 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:43.211117 23419 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20250116 02:59:43.228322 23507 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250116 02:59:43.262751 23417 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:43.263870 23417 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:43.264282 23417 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:43.236459 23273 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:43.242451 23368 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:43.273722 20571 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:43.245721 23430 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:43.280507 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 46011us
W20250116 02:59:43.283121 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 50361.9us
I20250116 02:59:43.235934 23311 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:43.300623 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 145193us
W20250116 02:59:43.301865 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 137020us
I20250116 02:59:43.285633 23335 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:43.311849 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.312378 23357 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:43.318758 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 97009.4us
W20250116 02:59:43.319739 20647 leader_election.cc:336] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.116s (SENT)
I20250116 02:59:43.320165 20647 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:43.328446 23357 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:43.330495 22871 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:43.332295 20647 connection.cc:232] RPC call timeout handler was delayed by -1.0284020540000256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1761s for a response
I20250116 02:59:43.334781 20649 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:43.338042 20647 leader_election.cc:336] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.886s (SENT)
I20250116 02:59:43.340207 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.292706 23507 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:43.299186 23368 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:43.289175 22993 consensus_queue.cc:1046] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20250116 02:59:43.359508 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.360847 20749 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
W20250116 02:59:43.358794 21302 leader_election.cc:336] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 2.177s (SENT)
I20250116 02:59:43.352180 23501 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 02:59:43.355660 22753 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:43.359891 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 63286.8us
I20250116 02:59:43.268610 23398 consensus_queue.cc:260] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:43.360298 23510 log.cc:926] Time spent T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.248s user 0.000s sys 0.002s
W20250116 02:59:43.387046 23512 log.cc:926] Time spent T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.144s user 0.002s sys 0.005s
I20250116 02:59:43.390028 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:43.392262 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.392500 23507 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:43.395125 23507 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:43.396641 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:43.397473 20605 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:43.400058 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.401161 20752 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:43.402487 20756 raft_consensus.cc:2388] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:43.403744 23311 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:43.418087 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:43.421092 23398 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:43.425920 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:43.430809 20750 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:43.432337 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:43.364560 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:43.434725 20647 leader_election.cc:336] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.172s (SENT)
I20250116 02:59:43.435391 20647 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:43.443336 20572 leader_election.cc:400] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Vote denied by peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647) with higher term. Message: Invalid argument: T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6.
I20250116 02:59:43.443908 20572 leader_election.cc:403] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250116 02:59:43.456122 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.363322 23483 consensus_queue.cc:1035] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.097s
I20250116 02:59:43.492506 20608 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:43.501055 20679 raft_consensus.cc:2405] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250116 02:59:43.483417 23311 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 02:59:43.447139 21329 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
W20250116 02:59:43.510511 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 100804us
I20250116 02:59:43.513669 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:43.526297 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 154128us
I20250116 02:59:43.496264 23419 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:43.496167 23368 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647) with higher term. Message: Invalid argument: T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6.
W20250116 02:59:43.528440 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 172969us
I20250116 02:59:43.402906 22753 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:43.543335 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.544670 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:43.545416 21302 leader_election.cc:336] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 2.486s (SENT)
I20250116 02:59:43.546391 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.546547 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.547173 20751 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 2.
I20250116 02:59:43.547720 20755 raft_consensus.cc:2388] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:43.548664 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:43.550942 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 116311us
W20250116 02:59:43.551827 20647 leader_election.cc:336] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.177s (SENT)
I20250116 02:59:43.552255 20647 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:43.553294 22519 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:43.554507 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.548678 20751 raft_consensus.cc:1270] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250116 02:59:43.539297 20750 raft_consensus.cc:2371] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5. Re-sending same reply.
I20250116 02:59:43.555568 20751 pending_rounds.cc:77] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 2
W20250116 02:59:43.553673 20647 leader_election.cc:336] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.494s (SENT)
W20250116 02:59:43.554368 21302 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 2.518s (SENT) [suppressed 50 similar messages]
I20250116 02:59:43.568935 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:43.563395 20751 pending_rounds.cc:99] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250116 02:59:43.571949 20751 raft_consensus.cc:2883] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:43.572418 20753 raft_consensus.cc:1270] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250116 02:59:43.570632 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 180053us
I20250116 02:59:43.571481 20572 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:43.569061 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.570421 20757 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:43.579874 23368 raft_consensus.cc:2758] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20250116 02:59:43.581609 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 76194us
I20250116 02:59:43.587911 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.588917 20750 raft_consensus.cc:2388] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:43.594254 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:43.597949 23514 log.cc:926] Time spent T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.398s user 0.002s sys 0.002s
I20250116 02:59:43.569674 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:43.616830 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:43.627075 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:43.600749 23466 raft_consensus.cc:3049] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:43.627331 23430 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:43.631348 23466 raft_consensus.cc:738] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:43.638020 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 84394.7us
I20250116 02:59:43.643549 23483 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250116 02:59:43.635577 23524 log.cc:926] Time spent T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
W20250116 02:59:43.671653 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 113543us
W20250116 02:59:43.699718 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 94545.2us
W20250116 02:59:43.701386 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 52539.9us
W20250116 02:59:43.777074 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 61492.3us
I20250116 02:59:43.800974 20755 raft_consensus.cc:1235] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 5. Current term is 6. Ops: []
W20250116 02:59:43.792208 23479 consensus_meta.cc:287] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.808s user 0.001s sys 0.015s
I20250116 02:59:43.782196 22753 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:43.736855 23417 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:43.819257 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 106584us
W20250116 02:59:43.839969 21302 leader_election.cc:336] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.647s (SENT)
I20250116 02:59:43.856890 20605 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:43.862212 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:43.862980 20755 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:43.814987 23430 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:43.720577 23466 consensus_queue.cc:260] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:43.872562 20571 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:43.882676 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:43.815660 23479 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:43.889539 23483 consensus_queue.cc:1035] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250116 02:59:43.890970 20554 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet bf7916e71bc94e1eba4ab8fd75257e48
W20250116 02:59:43.891393 20554 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet bf7916e71bc94e1eba4ab8fd75257e48
I20250116 02:59:43.891871 20682 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:43.894778 23507 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250116 02:59:43.895152 23519 log.cc:926] Time spent T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.568s user 0.000s sys 0.007s
I20250116 02:59:43.895335 23507 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:43.895762 23507 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
W20250116 02:59:43.897846 23516 log.cc:926] Time spent T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.654s user 0.005s sys 0.000s
I20250116 02:59:43.903839 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:43.907791 20749 raft_consensus.cc:2463] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:43.913484 20571 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:43.915989 20601 raft_consensus.cc:1235] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:43.893836 23492 consensus_queue.cc:1046] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20250116 02:59:43.935334 23368 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:43.924408 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:43.927974 21329 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250116 02:59:43.931078 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:43.932236 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:43.933092 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:43.933580 20680 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250116 02:59:43.935489 20683 raft_consensus.cc:1270] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:59:43.936102 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:43.923915 23311 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:43.938239 20649 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:43.937132 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:43.951996 20682 raft_consensus.cc:2388] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:43.915944 23483 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:43.951828 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:43.953298 20605 raft_consensus.cc:2388] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:43.949725 20604 raft_consensus.cc:2371] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6. Re-sending same reply.
I20250116 02:59:43.955948 20602 raft_consensus.cc:2388] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:43.956420 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:43.957669 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:43.959275 21335 raft_consensus.cc:2363] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:43.958389 20607 raft_consensus.cc:3054] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 6
I20250116 02:59:43.947466 21329 pending_rounds.cc:77] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Aborting all ops after (but not including) 5
I20250116 02:59:43.958652 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:43.963457 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:43.939630 23368 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:43.931316 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:43.961064 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:43.949798 23438 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:43.978931 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:43.963891 20649 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:43.979439 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:43.991427 20679 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:43.991477 20600 raft_consensus.cc:1270] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:43.964351 21329 pending_rounds.cc:99] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250116 02:59:43.992554 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:44.030372 20755 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:44.033753 20608 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:44.029332 20605 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20250116 02:59:43.991485 20723 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.808s (SENT)
W20250116 02:59:44.032572 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 60632.8us
I20250116 02:59:43.954660 23483 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:44.000437 23517 log.cc:926] Time spent T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.675s user 0.006s sys 0.001s
I20250116 02:59:43.968384 23526 consensus_queue.cc:1046] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20250116 02:59:43.969928 23368 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:43.967567 23466 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:43.999858 23417 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:43.991657 21327 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:44.046435 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:44.110845 20755 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:44.047791 20604 raft_consensus.cc:2388] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:44.093760 21300 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:44.047425 20607 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
W20250116 02:59:44.036022 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 44602.5us
I20250116 02:59:44.029335 20677 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:44.032085 23438 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:44.117022 23438 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:44.033308 21329 raft_consensus.cc:2883] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:43.964497 23311 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:44.088980 23535 log.cc:926] Time spent T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
I20250116 02:59:44.035463 23107 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:44.128835 23107 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:44.050573 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 69194us
I20250116 02:59:44.130190 20606 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:44.129618 23311 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:44.135044 23107 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:44.135932 23107 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:44.092926 23317 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:44.082080 23483 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:44.099006 22993 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:44.136580 23107 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:44.145088 20601 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20250116 02:59:44.148914 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 185123us
I20250116 02:59:44.151059 20679 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:44.148499 21299 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:44.154832 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 63505.8us
I20250116 02:59:44.141567 23541 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:44.142089 23479 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:44.106545 23507 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.145476 23317 raft_consensus.cc:695] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:44.157709 23398 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:44.178184 20571 leader_election.cc:400] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:44.178810 20571 leader_election.cc:403] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250116 02:59:44.143249 23419 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20250116 02:59:44.182173 23398 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:44.158761 22993 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.185535 23541 raft_consensus.cc:695] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:44.204779 23545 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
I20250116 02:59:44.210758 23541 consensus_queue.cc:237] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.215276 23507 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:44.222138 20755 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:44.226859 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:44.228461 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:44.163848 23438 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.231642 23540 raft_consensus.cc:3049] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:44.232223 23540 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:44.233146 23540 consensus_queue.cc:260] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.236192 23532 raft_consensus.cc:3049] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:44.236644 23532 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:44.236994 20682 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8. Ops: []
I20250116 02:59:44.237603 23532 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.240516 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:44.240749 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:44.244091 23337 consensus_queue.cc:1046] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 02:59:44.245114 23337 raft_consensus.cc:3049] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Stepping down as leader of term 7
I20250116 02:59:44.245460 23337 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:44.248566 23107 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:44.198662 23398 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:44.252218 23540 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:44.174836 20677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:44.233074 23438 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:44.266722 20571 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:44.275000 20604 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250116 02:59:44.258563 23507 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:44.264312 23532 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:44.246318 23337 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.253245 23419 consensus_queue.cc:1046] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20250116 02:59:44.294745 23553 consensus_queue.cc:1035] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20250116 02:59:44.227007 23317 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.345414 23546 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:44.281544 23438 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:44.302927 23368 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.342600 23311 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:44.387678 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:44.375123 22993 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:44.411638 23337 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
W20250116 02:59:44.411118 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 114020us
I20250116 02:59:44.352833 23438 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:44.435894 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 174114us
W20250116 02:59:44.436966 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 47350.9us
I20250116 02:59:44.440932 20649 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:44.441345 20571 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:44.474651 23311 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:44.507622 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:44.478193 23561 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250116 02:59:44.526810 23368 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:44.528218 23438 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:44.472770 23338 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20250116 02:59:44.536859 20503 catalog_manager.cc:5526] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:59:44.475349 23483 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.544106 23483 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:44.545121 23558 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 02:59:44.546820 20599 raft_consensus.cc:1235] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:44.552955 21300 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:44.553210 20600 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250116 02:59:44.566920 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 119698us
I20250116 02:59:44.582141 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:44.583060 20608 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:44.584383 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:44.591882 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:44.597234 20602 raft_consensus.cc:1235] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20250116 02:59:44.575366 23417 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election won for term 5
W20250116 02:59:44.591388 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 92867.7us
I20250116 02:59:44.581225 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:44.611054 21331 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:44.611716 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:44.612699 20608 raft_consensus.cc:1270] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:44.614924 23438 consensus_queue.cc:1035] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250116 02:59:44.622267 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 47748.6us
I20250116 02:59:44.627524 20677 raft_consensus.cc:604] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:44.624330 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:44.630704 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:44.634330 20683 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:44.652243 20646 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:44.653992 20571 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:44.623430 23417 raft_consensus.cc:695] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:44.690552 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:44.694907 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:44.695822 20676 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:44.635310 23507 consensus_queue.cc:1035] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250116 02:59:44.673270 22791 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:44.704771 20571 leader_election.cc:400] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:44.706918 20571 leader_election.cc:403] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250116 02:59:44.707321 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:44.710563 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:44.711357 23572 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:44.712008 23572 raft_consensus.cc:695] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:44.712260 23483 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 5
I20250116 02:59:44.676844 23565 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:44.711405 20679 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
W20250116 02:59:44.714180 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 45573.5us
I20250116 02:59:44.716022 20608 raft_consensus.cc:2388] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:44.717164 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:44.705945 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:44.715353 22791 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:44.734170 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:44.705816 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:44.742069 20602 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:44.743819 21333 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:44.743676 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:44.729084 23368 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:44.699726 23398 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.756758 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:44.740608 23386 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:44.745469 22791 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:44.753926 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:44.767243 23574 log.cc:926] Time spent T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20250116 02:59:44.727188 23571 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20250116 02:59:44.770160 23572 consensus_queue.cc:237] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:44.752219 23368 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:44.761268 23417 consensus_queue.cc:237] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.776494 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:44.747797 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:44.780712 21332 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:44.766985 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:44.771803 23570 consensus_queue.cc:1035] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250116 02:59:44.788604 20676 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:44.788367 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:44.794713 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:44.751871 23438 consensus_queue.cc:1035] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20250116 02:59:44.811813 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:44.785564 23107 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:44.741286 23565 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:44.825120 20571 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:44.797258 23368 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:44.828316 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:44.832280 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:44.835484 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:44.838224 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:44.819691 21331 raft_consensus.cc:3054] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:44.827917 23311 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250116 02:59:44.861330 23564 raft_consensus.cc:2798] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:44.813534 23507 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20250116 02:59:44.787349 23386 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:44.859508 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:44.830893 23107 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:44.869355 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 83983.2us
I20250116 02:59:44.975894 23107 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:44.976915 23107 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:44.979101 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:44.979983 20601 raft_consensus.cc:2388] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:44.853133 21327 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:44.976511 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:44.981489 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:44.982631 20603 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 6
I20250116 02:59:44.978031 23107 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:45.023361 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:44.981197 23317 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:45.034354 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 205084us
I20250116 02:59:45.047046 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:45.112319 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 58814.5us
W20250116 02:59:45.115005 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 81647.9us
I20250116 02:59:45.116729 20571 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:45.129539 23564 raft_consensus.cc:695] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:45.131043 23546 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:45.131547 23546 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:45.131897 23546 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:45.041819 23317 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:45.137624 23317 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:45.140499 23386 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:45.143270 21335 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:45.143440 21332 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:45.140237 23302 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20250116 02:59:45.148645 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 111909us
I20250116 02:59:45.152308 20571 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:45.180146 23589 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:45.180980 23589 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.113181 23592 consensus_queue.cc:1046] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.126s
W20250116 02:59:45.183070 20720 connection.cc:232] RPC call timeout handler was delayed by -2.5621408840000868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
W20250116 02:59:45.190111 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 49199.5us
W20250116 02:59:45.183588 20720 connection.cc:232] RPC call timeout handler was delayed by -2.4721408840000549s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20250116 02:59:45.194140 20720 connection.cc:232] RPC call timeout handler was delayed by -2.4100408840000682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
W20250116 02:59:45.194509 20720 connection.cc:232] RPC call timeout handler was delayed by -2.3290408840000509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17900000000000002s for a response
W20250116 02:59:45.194898 20720 connection.cc:232] RPC call timeout handler was delayed by -2.2894408839999869s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
I20250116 02:59:45.114379 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:45.199237 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:45.201129 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:45.183059 23558 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:45.211879 20720 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:45.210844 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 231958us
I20250116 02:59:45.211710 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:45.181042 23107 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:45.216465 23590 log.cc:926] Time spent T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250116 02:59:45.219699 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:45.220690 20603 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:45.217259 20748 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:45.234248 20680 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250116 02:59:45.202431 23549 raft_consensus.cc:3049] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:45.239111 23549 raft_consensus.cc:738] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:45.240177 23549 consensus_queue.cc:260] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:45.243048 23596 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 02:59:45.248803 23107 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:45.249576 23107 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:45.250839 20572 connection.cc:232] RPC call timeout handler was delayed by -2.4557477010000639s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
W20250116 02:59:45.251343 20572 connection.cc:232] RPC call timeout handler was delayed by -2.3765477010000495s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1636s for a response
W20250116 02:59:45.251644 20572 connection.cc:232] RPC call timeout handler was delayed by -2.3315477010000905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
W20250116 02:59:45.251945 20572 connection.cc:232] RPC call timeout handler was delayed by -2.23614770100005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17920000000000003s for a response
W20250116 02:59:45.252238 20572 connection.cc:232] RPC call timeout handler was delayed by -2.0984477010000546s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
W20250116 02:59:45.252559 20572 connection.cc:232] RPC call timeout handler was delayed by -2.0219477010000446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20299999999999999s for a response
W20250116 02:59:45.252835 20572 connection.cc:232] RPC call timeout handler was delayed by -1.9085477010000886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
W20250116 02:59:45.253104 20572 connection.cc:232] RPC call timeout handler was delayed by -1.805047701000035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2271s for a response
W20250116 02:59:45.253402 20572 connection.cc:232] RPC call timeout handler was delayed by -1.1570477010000104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29910000000000003s for a response
W20250116 02:59:45.253683 20572 connection.cc:232] RPC call timeout handler was delayed by -1.0706477010000981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30870000000000003s for a response
I20250116 02:59:45.217536 23565 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:45.255088 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.197903 23564 consensus_queue.cc:237] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:45.249449 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 47963.8us
I20250116 02:59:45.261512 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.293360 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.292778 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.321915 20720 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:45.292889 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:45.264153 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 45019.3us
I20250116 02:59:45.299749 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:45.272565 23483 consensus_meta.cc:287] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.560s user 0.012s sys 0.000s
I20250116 02:59:45.321780 20599 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:45.337005 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:45.251120 23558 raft_consensus.cc:695] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:45.306959 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:45.342305 20680 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:45.313503 23368 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:45.315012 23583 log.cc:926] Time spent T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20250116 02:59:45.251772 23107 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:45.338035 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.294726 23592 consensus_queue.cc:1225] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Peer f65e8d7f76144fab9725ca32a6328c3f log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250116 02:59:45.349112 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48834 (request call id 31) took 1186 ms. Trace:
I20250116 02:59:45.349633 20677 rpcz_store.cc:276] 0116 02:59:44.155149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:44.155941 (+ 792us) service_pool.cc:225] Handling call
0116 02:59:45.341826 (+1185885us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54000870d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d09bc3 00007fc32bc7082a 00007fc325349d14 00007fc325349ca2 00007fc325349bd1 00007fc325349b4c 00007fc325348e02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:45.341919 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":842827520}
I20250116 02:59:45.346405 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.351397 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.354166 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.354871 20601 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:45.355525 23602 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:45.347548 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 297861us
W20250116 02:59:45.313565 22791 consensus_meta.cc:287] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.522s user 0.016s sys 0.002s
I20250116 02:59:45.355784 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:45.495827 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 112324us
W20250116 02:59:45.418216 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 72740.4us
I20250116 02:59:45.419180 20608 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
W20250116 02:59:45.382853 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 58517.8us
I20250116 02:59:45.460232 20680 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:45.355008 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.356997 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.442915 23607 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250116 02:59:45.447403 20647 leader_election.cc:336] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 4.805s (SENT)
I20250116 02:59:45.504664 22791 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.506132 21302 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.344926 23492 consensus_queue.cc:1035] T 49f9e38171174b70b4373b56dbcd90be P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.162s
I20250116 02:59:45.497273 20720 leader_election.cc:400] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5.
I20250116 02:59:45.504076 20647 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.416033 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.540689 21302 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.342199 23483 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:45.505041 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.356015 23589 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.551348 23553 raft_consensus.cc:2743] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:45.552055 23368 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:45.552608 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 134307us
I20250116 02:59:45.360361 21331 raft_consensus.cc:2463] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
W20250116 02:59:45.553824 20647 leader_election.cc:336] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 4.945s (SENT)
I20250116 02:59:45.554108 21302 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.357568 23107 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:45.555125 21327 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:45.554922 23107 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.555662 23492 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:45.555944 21302 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.415446 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:45.557297 20647 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.557509 23466 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:45.429127 23592 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:45.559008 21302 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.556087 23605 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250116 02:59:45.565140 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 50179us
W20250116 02:59:45.566946 20647 leader_election.cc:336] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.667s (SENT)
I20250116 02:59:45.567533 20647 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:45.446482 23577 log.cc:926] Time spent T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.735s user 0.000s sys 0.007s
I20250116 02:59:45.572196 21302 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.576825 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:45.571564 23602 consensus_queue.cc:1046] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250116 02:59:45.587973 20723 leader_election.cc:336] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.535s (SENT)
I20250116 02:59:45.583683 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:45.583046 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.587322 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.561398 23337 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250116 02:59:45.591805 20723 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 3.228s (SENT)
I20250116 02:59:45.561198 23595 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:45.566758 23607 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:45.595520 20723 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.604099 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:45.602699 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.580595 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:45.581907 23596 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:45.586946 23302 raft_consensus.cc:2743] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:45.570396 23438 consensus_queue.cc:1046] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
W20250116 02:59:45.611243 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 51312.6us
I20250116 02:59:45.590920 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:45.595964 23483 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:45.622890 23313 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:45.623826 23438 raft_consensus.cc:3049] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:45.623529 23313 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.624259 23438 raft_consensus.cc:738] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:45.624415 23559 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:45.611990 23607 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:45.625934 23433 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20250116 02:59:45.498016 23546 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.612726 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.558744 23592 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:45.614853 20647 leader_election.cc:336] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.712s (SENT)
I20250116 02:59:45.618917 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:45.629839 20647 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.630026 21332 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:45.620800 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:45.618355 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.609705 23564 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:45.598415 23335 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:45.630824 20607 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:45.633416 23564 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.638232 23592 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:45.646466 20647 leader_election.cc:336] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.761s (SENT)
I20250116 02:59:45.647030 20647 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.647487 23564 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.647938 23107 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.649724 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:45.650269 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.650462 21328 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 02:59:45.650723 23565 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:45.650507 20605 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
W20250116 02:59:45.643324 20572 leader_election.cc:336] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.945s (SENT)
I20250116 02:59:45.652737 20572 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:45.653987 20572 leader_election.cc:336] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.030s (SENT)
W20250116 02:59:45.651904 20647 leader_election.cc:336] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.801s (SENT)
I20250116 02:59:45.625422 23599 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:45.625041 23559 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.651836 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:45.656816 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.657271 23559 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.639531 23603 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:45.658097 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.658823 23368 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:45.658358 23603 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.659206 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.655361 23599 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:45.660749 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:45.662111 23599 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:45.443784 23558 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.642052 23546 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.642976 21329 raft_consensus.cc:2388] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:45.664081 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:45.607347 22791 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.643170 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:45.664785 21328 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:45.665050 21331 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:45.623070 23483 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.608450 23602 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:45.668188 23483 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:45.632028 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 134921us
I20250116 02:59:45.671036 20720 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.672353 20646 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:45.672845 20720 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.673619 23585 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:45.654788 21327 raft_consensus.cc:2363] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 7. Current term is 8.
I20250116 02:59:45.674139 23585 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:45.674669 23585 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:45.675711 23438 consensus_queue.cc:260] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.645301 23313 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:45.681234 20503 catalog_manager.cc:5526] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:45.683590 23599 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:45.684688 20647 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:45.659296 23368 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:45.688192 23368 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:45.691284 22791 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250116 02:59:45.692648 20647 leader_election.cc:336] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.843s (SENT)
I20250116 02:59:45.692976 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.715734 20608 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:45.716008 20647 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:45.715674 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 61753.3us
W20250116 02:59:45.717185 20572 leader_election.cc:336] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.156s (SENT)
I20250116 02:59:45.717785 20572 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:45.717011 23570 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:45.719234 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.720235 20608 raft_consensus.cc:2405] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250116 02:59:45.721257 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.699154 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:45.707473 20571 leader_election.cc:400] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 7. Current term is 8.
I20250116 02:59:45.727193 20607 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:45.683099 20503 catalog_manager.cc:5526] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 3 to 4, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 4 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } }
I20250116 02:59:45.710351 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:45.711261 23483 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:45.711846 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:45.736099 20649 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:45.737092 20683 raft_consensus.cc:2388] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:45.716997 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:45.687439 23603 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.723518 20502 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48594 (request call id 218) took 9043 ms. Trace:
I20250116 02:59:45.735872 21332 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:45.744423 20603 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20250116 02:59:45.752080 20572 leader_election.cc:336] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.271s (SENT)
I20250116 02:59:45.752482 21327 raft_consensus.cc:2405] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20250116 02:59:45.753019 20572 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:45.746734 20675 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:45.754423 20572 leader_election.cc:336] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.991s (SENT)
I20250116 02:59:45.752919 20503 catalog_manager.cc:5526] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 4, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:45.754868 20572 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:45.756078 23548 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:45.755620 20503 catalog_manager.cc:5526] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 2, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 2 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } }
I20250116 02:59:45.756870 23548 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:45.756503 23585 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.742486 23559 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:45.726713 23546 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:45.723800 23368 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.727375 20571 leader_election.cc:403] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250116 02:59:45.655459 23564 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:45.699213 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:45.767670 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 60356.4us
I20250116 02:59:45.662191 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.732450 20680 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:45.732488 20608 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:45.775938 20599 raft_consensus.cc:2405] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250116 02:59:45.777138 20647 leader_election.cc:336] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.877s (SENT)
I20250116 02:59:45.777652 20647 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:45.763931 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:45.769114 23507 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:45.769477 20503 catalog_manager.cc:5526] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 2 to 3, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 3 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:45.775751 20680 raft_consensus.cc:604] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.780038 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:45.780722 20603 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:45.778765 21332 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
W20250116 02:59:45.790567 20723 leader_election.cc:336] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.854s (SENT)
I20250116 02:59:45.758961 23585 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.743690 21331 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:45.735092 23483 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.803491 20503 catalog_manager.cc:5526] T 5ba18c5d216643cbb5689397ab22da23 P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 1 to 3, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
W20250116 02:59:45.807534 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 45571.6us
W20250116 02:59:45.808375 20647 leader_election.cc:336] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.102s (SENT)
I20250116 02:59:45.780378 23559 raft_consensus.cc:695] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:45.816745 20603 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:45.819904 20503 catalog_manager.cc:5526] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 3 to 4, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 4 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } }
I20250116 02:59:45.826992 21332 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:45.826673 20503 catalog_manager.cc:5526] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:45.828214 20646 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:45.775014 23603 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20250116 02:59:45.797833 23546 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:45.805856 23561 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:45.828375 20503 catalog_manager.cc:5526] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb reported cstate change: term changed from 3 to 6, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 6 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:45.837271 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:45.841042 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.845085 20607 raft_consensus.cc:2388] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 8: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
W20250116 02:59:45.847975 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 76461.9us
I20250116 02:59:45.832715 23581 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election won for term 8
I20250116 02:59:45.855021 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 482) took 1351 ms. Trace:
I20250116 02:59:45.855372 20600 rpcz_store.cc:276] 0116 02:59:44.503199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:44.503349 (+ 150us) service_pool.cc:225] Handling call
0116 02:59:44.503745 (+ 396us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:44.554004 (+ 50259us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:45.852823 (+1298819us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b54000564d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:45.854941 (+ 2118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113148288}
I20250116 02:59:45.856323 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.857115 20607 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:45.846841 21327 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20250116 02:59:45.859150 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 186397us
I20250116 02:59:45.862066 20720 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.866892 20720 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.867484 20554 placement_policy_util.cc:244] tablet bf7916e71bc94e1eba4ab8fd75257e48: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
W20250116 02:59:45.867914 23573 log.cc:926] Time spent T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 1.157s user 0.015s sys 0.025s
I20250116 02:59:45.848531 23553 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:45.740835 20502 rpcz_store.cc:276] 0116 02:59:36.679936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:36.701575 (+ 21639us) service_pool.cc:225] Handling call
0116 02:59:45.723404 (+9021829us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:42.435320 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:42.435796 (+ 476us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:42.435831 (+ 35us) write_op.cc:435] Acquired schema lock
0116 02:59:42.435846 (+ 15us) tablet.cc:659] Decoding operations
0116 02:59:42.437465 (+ 1619us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:42.437573 (+ 108us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:42.437602 (+ 29us) tablet.cc:682] Acquiring locks for 14 operations
0116 02:59:42.439100 (+ 1498us) tablet.cc:698] Row locks acquired
0116 02:59:42.439118 (+ 18us) write_op.cc:260] PREPARE: finished
0116 02:59:42.439308 (+ 190us) write_op.cc:270] Start()
0116 02:59:42.439444 (+ 136us) write_op.cc:276] Timestamp: P: 1736996382439252 usec, L: 0
0116 02:59:42.439460 (+ 16us) op_driver.cc:348] REPLICATION: starting
0116 02:59:42.440296 (+ 836us) log.cc:844] Serialized 5648 byte log entry
0116 02:59:42.596544 (+156248us) op_driver.cc:464] REPLICATION: finished
0116 02:59:42.926850 (+330306us) write_op.cc:301] APPLY: starting
0116 02:59:42.943289 (+ 16439us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:43.046030 (+102741us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:43.087750 (+ 41720us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:43.324641 (+236891us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:43.325084 (+ 443us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0116 02:59:43.325233 (+ 149us) write_op.cc:312] APPLY: finished
0116 02:59:43.325901 (+ 668us) log.cc:844] Serialized 101 byte log entry
0116 02:59:43.326169 (+ 268us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:43.342732 (+ 16563us) write_op.cc:454] Released schema lock
0116 02:59:43.352753 (+ 10021us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:43.413729 (+ 60976us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:43.413769 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":90,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":280317,"apply.run_cpu_time_us":12248,"apply.run_wall_time_us":528473,"num_ops":14,"prepare.queue_time_us":489198,"prepare.run_cpu_time_us":6822,"prepare.run_wall_time_us":11059,"raft.queue_time_us":1333,"raft.run_cpu_time_us":15498,"raft.run_wall_time_us":228705,"replication_time_us":156669,"thread_start_us":844792,"threads_started":4,"wal-append.queue_time_us":6409,"wal-append.run_cpu_time_us":6224,"wal-append.run_wall_time_us":1880828}]]}
W20250116 02:59:45.872565 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 73026us
I20250116 02:59:45.881105 21302 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:45.864491 23581 raft_consensus.cc:695] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:45.851771 23546 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:45.883479 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 45474.6us
I20250116 02:59:45.873046 23592 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:45.888056 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.889637 20720 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.880285 23107 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:45.893095 20607 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:45.883575 22753 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:45.914801 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.917568 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2228) took 1487 ms. Trace:
I20250116 02:59:45.911545 22753 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:45.922508 22753 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:45.923458 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.919153 20604 rpcz_store.cc:276] 0116 02:59:44.429806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:44.430106 (+ 300us) service_pool.cc:225] Handling call
0116 02:59:44.439169 (+ 9063us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:44.458045 (+ 18876us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:44.458081 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:44.473075 (+ 14994us) log.cc:844] Serialized 26 byte log entry
0116 02:59:44.586986 (+113911us) raft_consensus.cc:1640] Marking committed up to 1
0116 02:59:44.587029 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:44.587106 (+ 77us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:45.833622 (+1246516us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:45.913878 (+ 80256us) raft_consensus.cc:1680] finished
0116 02:59:45.913910 (+ 32us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:45.917487 (+ 3577us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7661,"raft.queue_time_us":15491,"raft.run_cpu_time_us":481,"raft.run_wall_time_us":43945,"spinlock_wait_cycles":5068160,"thread_start_us":66238,"threads_started":1,"wal-append.queue_time_us":205414}
I20250116 02:59:45.925050 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.909535 23368 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.928004 20607 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:45.929294 23601 raft_consensus.cc:3049] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:45.929502 20607 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:45.929736 23601 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:45.930752 23601 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.864904 23559 consensus_queue.cc:237] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:45.883973 23553 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:45.932698 20647 leader_election.cc:336] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.126s (SENT)
I20250116 02:59:45.933179 20647 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:45.928901 23624 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20250116 02:59:45.913636 23592 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:45.937863 20720 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.891019 23623 consensus_queue.cc:1035] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
W20250116 02:59:45.949111 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 45023.3us
I20250116 02:59:45.935102 23483 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:45.957626 20720 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:45.942709 23601 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:45.961899 23546 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:45.962106 20723 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:45.963886 20722 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.013s sys 0.026s
I20250116 02:59:45.963688 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:45.965142 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:45.965766 23398 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:45.973148 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:45.974021 20674 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:45.976492 20720 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:45.976796 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:45.964223 20722 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.698s user 0.014s sys 0.027s
I20250116 02:59:45.955730 23107 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:45.988909 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:45.992933 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:45.993562 20682 raft_consensus.cc:2388] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 8: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:45.990782 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:45.965462 23580 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:46.001582 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.006271 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.007043 20755 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:45.998430 20675 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:46.010515 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.011296 20754 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:46.012086 20750 raft_consensus.cc:1270] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:59:46.014833 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:46.019338 20754 raft_consensus.cc:2388] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:46.020139 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.020937 20749 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:46.021736 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.022715 20757 raft_consensus.cc:2405] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250116 02:59:46.005054 23602 consensus_queue.cc:1046] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20250116 02:59:45.988639 23592 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:45.954509 23581 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:46.023643 20647 leader_election.cc:336] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.504s (SENT)
I20250116 02:59:46.030377 23602 consensus_queue.cc:1225] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [LEADER]: Peer f65e8d7f76144fab9725ca32a6328c3f log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250116 02:59:46.016232 23107 consensus_queue.cc:1046] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20250116 02:59:46.027726 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.018697 21299 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:46.029805 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:46.008085 23599 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:46.032775 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:46.051594 20682 raft_consensus.cc:1270] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250116 02:59:46.050067 20752 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:46.055310 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.067144 20754 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:46.068198 20753 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:46.051074 23602 raft_consensus.cc:3049] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:46.067054 20674 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 02:59:46.036927 23507 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 7. Current term is 8.
I20250116 02:59:46.074589 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.087431 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.088771 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.089524 20753 raft_consensus.cc:2388] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:46.071513 23107 consensus_queue.cc:1225] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [LEADER]: Peer f65e8d7f76144fab9725ca32a6328c3f log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250116 02:59:46.039666 23619 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:46.091547 23619 raft_consensus.cc:695] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:46.092423 23619 consensus_queue.cc:237] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:46.094494 23592 raft_consensus.cc:3049] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:46.095082 23592 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:46.095992 23592 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:46.098883 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.099428 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:46.100035 20753 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:46.100350 20752 raft_consensus.cc:2388] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:46.030973 23611 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:46.079586 22753 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:46.077378 23602 raft_consensus.cc:738] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:46.124214 23602 consensus_queue.cc:260] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:46.125645 23602 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:46.126040 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.111375 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:46.115687 20647 leader_election.cc:336] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.763s (SENT)
I20250116 02:59:46.112684 20603 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
W20250116 02:59:46.089046 20605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53964 (request call id 492) took 1257 ms (client timeout 1584 ms). Trace:
W20250116 02:59:46.129482 20605 rpcz_store.cc:269] 0116 02:59:44.831288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:44.832022 (+ 734us) service_pool.cc:225] Handling call
0116 02:59:46.088868 (+1256846us) spinlock_profiling.cc:243] Waited 748 ms on lock 0x7b540014d2d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:46.088967 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1719272832}
I20250116 02:59:46.105480 21299 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:46.065596 23616 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 02:59:46.131587 23594 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election won for term 8
I20250116 02:59:46.131871 23553 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:46.132161 23594 raft_consensus.cc:695] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:46.126782 20750 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:46.120692 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.139394 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.102569 23438 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:46.170203 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:46.171010 20754 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:46.171283 21332 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20250116 02:59:46.171737 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.172395 20752 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:46.066009 23599 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:46.197415 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 211672us
W20250116 02:59:46.198673 20723 leader_election.cc:336] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.900s (SENT)
I20250116 02:59:46.198567 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.201416 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:46.206776 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.207358 20750 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:46.204344 20680 consensus_queue.cc:237] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:46.210961 23594 consensus_queue.cc:237] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:46.195616 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:46.224987 23617 log.cc:926] Time spent T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.402s user 0.000s sys 0.001s
I20250116 02:59:46.226007 22753 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:46.226915 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:46.227195 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.230811 23613 raft_consensus.cc:2758] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20250116 02:59:46.232753 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:46.233652 20676 raft_consensus.cc:2388] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 8: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:46.235088 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2231) took 1660 ms. Trace:
I20250116 02:59:46.235502 20606 rpcz_store.cc:276] 0116 02:59:44.575063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:44.575531 (+ 468us) service_pool.cc:225] Handling call
0116 02:59:45.853277 (+1277746us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:45.987034 (+133757us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:45.987068 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:46.027437 (+ 40369us) log.cc:844] Serialized 26 byte log entry
0116 02:59:46.134316 (+106879us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:46.134357 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:46.134442 (+ 85us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:46.232944 (+ 98502us) raft_consensus.cc:1680] finished
0116 02:59:46.232966 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:46.234812 (+ 1846us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54000564d0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:46.235026 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5365,"raft.queue_time_us":64247,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":17261,"spinlock_wait_cycles":2893873408,"thread_start_us":226999,"threads_started":2,"wal-append.queue_time_us":192025}
I20250116 02:59:46.236902 20754 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
W20250116 02:59:46.237684 23634 log.cc:926] Time spent T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250116 02:59:46.239744 23599 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:46.241783 23592 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:46.243104 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:46.255329 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:46.259630 20753 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:46.261423 21335 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250116 02:59:46.266100 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.272084 23565 consensus_queue.cc:1046] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 02:59:46.273175 23561 raft_consensus.cc:3049] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:46.273548 23561 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:46.274427 23561 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:46.276212 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.277130 20754 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:46.257714 23611 raft_consensus.cc:695] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:46.287911 23338 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:46.288715 23338 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:46.289727 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:46.290581 20750 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:46.291219 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.291891 20754 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:46.309759 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.293392 23561 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:46.323894 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:46.332803 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.333828 20753 raft_consensus.cc:2371] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3. Re-sending same reply.
I20250116 02:59:46.334311 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.331367 23579 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:46.324518 23338 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:46.351940 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.353147 20753 raft_consensus.cc:1270] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250116 02:59:46.379323 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:46.381179 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:46.382067 20756 raft_consensus.cc:2371] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3. Re-sending same reply.
I20250116 02:59:46.407122 20753 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:46.342092 23611 consensus_queue.cc:237] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:46.431018 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.435204 20750 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:46.443405 23607 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:46.423506 23606 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.027s
I20250116 02:59:46.464756 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.468187 20754 raft_consensus.cc:2405] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20250116 02:59:46.479411 23648 log.cc:926] Time spent T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20250116 02:59:46.480849 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1706) took 1722 ms. Trace:
I20250116 02:59:46.481263 20679 rpcz_store.cc:276] 0116 02:59:44.758743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:44.758916 (+ 173us) service_pool.cc:225] Handling call
0116 02:59:44.773074 (+ 14158us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:44.773500 (+ 426us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:44.773526 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:46.139691 (+1366165us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b2c0005b468. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc3256f0741 00007fc3256ffd72 00007fc326d4b141 00007fc326d0f58a 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782
0116 02:59:46.140161 (+ 470us) log.cc:844] Serialized 26 byte log entry
0116 02:59:46.329608 (+189447us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:46.329647 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:46.329845 (+ 198us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:46.480517 (+150672us) raft_consensus.cc:1680] finished
0116 02:59:46.480542 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:46.480781 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13929,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":158,"spinlock_wait_cycles":25402624,"thread_start_us":152344,"threads_started":1,"wal-append.queue_time_us":222845}
I20250116 02:59:46.487696 23558 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:46.488504 23558 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:46.491056 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:46.491611 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 437994us
I20250116 02:59:46.494699 20605 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:46.496551 20680 raft_consensus.cc:1270] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250116 02:59:46.497908 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:46.515426 20647 leader_election.cc:336] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.180s (SENT)
W20250116 02:59:46.697799 20572 leader_election.cc:336] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.720s (SENT)
I20250116 02:59:46.711372 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:46.690912 23654 log.cc:926] Time spent T 00000000000000000000000000000000 P 65b617bfeca24d2cadbd950d2ebf10d8: Append to log took a long time: real 0.202s user 0.000s sys 0.004s
W20250116 02:59:46.702868 23573 log.cc:926] Time spent T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.201s user 0.000s sys 0.000s
I20250116 02:59:46.736836 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:46.710602 23657 log.cc:926] Time spent T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.223s user 0.004s sys 0.000s
I20250116 02:59:46.765420 20571 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:46.782078 20754 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
W20250116 02:59:46.770438 21331 consensus_meta.cc:287] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.527s user 0.010s sys 0.003s
I20250116 02:59:46.788653 21331 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
W20250116 02:59:46.789350 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 273933us
I20250116 02:59:46.701570 23492 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20250116 02:59:46.702097 23626 consensus_queue.cc:1035] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.293s
I20250116 02:59:46.954475 20604 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250116 02:59:46.847992 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 157369us
I20250116 02:59:46.817943 20601 raft_consensus.cc:1235] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
W20250116 02:59:46.957747 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 259984us
I20250116 02:59:46.817535 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.990949 21335 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250116 02:59:46.962865 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.961266 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:46.989137 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 02:59:47.065279 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 76254.8us
W20250116 02:59:46.976042 23662 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20250116 02:59:47.002421 23622 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.100s
I20250116 02:59:46.958668 23608 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:47.068221 23659 log.cc:926] Time spent T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.065s user 0.001s sys 0.002s
I20250116 02:59:46.787983 23636 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:47.105827 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:47.111826 20647 leader_election.cc:336] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.525s (SENT)
I20250116 02:59:47.002832 23368 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:47.112591 20647 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:47.069878 23663 log.cc:926] Time spent T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.103s user 0.002s sys 0.000s
I20250116 02:59:47.110662 23636 raft_consensus.cc:695] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:47.086464 23625 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election won for term 8
I20250116 02:59:47.119550 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.123440 20757 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:47.103900 23579 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:47.130402 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.110395 23608 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:47.090054 23553 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
W20250116 02:59:46.828326 23615 log.cc:926] Time spent T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
W20250116 02:59:47.133463 20647 leader_election.cc:336] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.746s (SENT)
I20250116 02:59:47.145352 20647 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:47.131511 23625 raft_consensus.cc:695] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 8 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:47.080719 20572 leader_election.cc:336] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.794s (SENT)
W20250116 02:59:47.152338 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 71668.3us
I20250116 02:59:46.889046 23558 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:46.821460 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:47.135901 23438 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:47.085122 20571 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:47.164397 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 79348.5us
I20250116 02:59:47.160722 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.133572 23579 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:47.122139 23368 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:47.174685 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.175086 20757 raft_consensus.cc:2388] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 4: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:47.175480 20749 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:47.167563 23671 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:47.184599 21335 raft_consensus.cc:1270] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:47.184738 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.168521 23558 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 02:59:47.191617 20601 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 02:59:47.194393 20606 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:47.197347 20750 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:47.207885 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:47.208839 21331 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:47.211690 20602 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250116 02:59:47.214834 21331 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:47.238314 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.239383 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.240378 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.241215 20683 raft_consensus.cc:2388] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:47.191355 23623 consensus_queue.cc:1035] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
I20250116 02:59:47.260710 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.261023 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.261689 20680 raft_consensus.cc:2405] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250116 02:59:47.261461 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.259814 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.259742 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.263988 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.267006 21331 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250116 02:59:47.267257 20720 connection.cc:232] RPC call timeout handler was delayed by -2.0029725999999073s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
W20250116 02:59:47.267879 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 90981.7us
W20250116 02:59:47.267769 20720 connection.cc:232] RPC call timeout handler was delayed by -1.951672599999938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20250116 02:59:47.268855 20720 connection.cc:232] RPC call timeout handler was delayed by -1.9327725999999075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
W20250116 02:59:47.263653 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 71486.2us
W20250116 02:59:47.269141 20720 connection.cc:232] RPC call timeout handler was delayed by -1.8688725999999178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18080000000000002s for a response
W20250116 02:59:47.270382 20720 connection.cc:232] RPC call timeout handler was delayed by -1.7860725999998976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
W20250116 02:59:47.270733 20720 connection.cc:232] RPC call timeout handler was delayed by -1.7113725999998906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
W20250116 02:59:47.271000 20720 connection.cc:232] RPC call timeout handler was delayed by -1.6699725999999373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2029s for a response
W20250116 02:59:47.271268 20720 connection.cc:232] RPC call timeout handler was delayed by -1.5511725999999726s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21610000000000001s for a response
W20250116 02:59:47.271438 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 92192.4us
W20250116 02:59:47.271562 20720 connection.cc:232] RPC call timeout handler was delayed by -1.542172599999958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
W20250116 02:59:47.275593 20720 connection.cc:232] RPC call timeout handler was delayed by -1.426072599999884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22999999999999998s for a response
I20250116 02:59:47.289543 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.305671 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.306811 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:47.311013 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.313638 23609 consensus_queue.cc:1035] T ff78e82fe8da4dd58822fb1d565eb3be P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:59:47.315845 23579 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:47.318300 20754 raft_consensus.cc:1235] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250116 02:59:47.319180 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.324221 20757 raft_consensus.cc:2363] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5.
I20250116 02:59:47.330566 20750 raft_consensus.cc:2363] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 2. Current term is 3.
W20250116 02:59:47.331601 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 120593us
I20250116 02:59:47.333498 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:47.334223 21331 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:47.337805 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.339625 20754 raft_consensus.cc:2388] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:47.340447 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:47.341081 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 154521us
I20250116 02:59:47.343144 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:47.343581 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.344792 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:47.341310 20754 raft_consensus.cc:2405] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250116 02:59:47.347667 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:47.349363 20646 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:47.289975 23608 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:47.345338 20753 raft_consensus.cc:2371] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6. Re-sending same reply.
I20250116 02:59:47.191165 23636 consensus_queue.cc:237] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:47.389422 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:47.392836 20752 raft_consensus.cc:2405] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250116 02:59:47.396723 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.208537 23625 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:47.332796 20748 raft_consensus.cc:1270] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:59:47.400338 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:47.389948 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.325054 23681 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:47.309299 23690 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.041s
I20250116 02:59:47.398527 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:47.408039 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:47.420713 20757 raft_consensus.cc:2388] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:47.410527 20572 leader_election.cc:336] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.548s (SENT)
I20250116 02:59:47.422536 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.427052 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:47.399675 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 60107.5us
W20250116 02:59:47.429441 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 160909us
I20250116 02:59:47.430202 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.264242 23632 consensus_queue.cc:1035] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
I20250116 02:59:47.402241 23608 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 02:59:47.437283 20720 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:47.435292 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.441954 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.424670 23553 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:47.424232 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.446707 20720 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:47.443684 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:47.463959 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 244660us
W20250116 02:59:47.465921 20647 leader_election.cc:336] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.638s (SENT)
I20250116 02:59:47.466665 20647 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:47.437395 23608 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:47.433454 23368 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:47.444212 23673 log.cc:926] Time spent T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.188s user 0.000s sys 0.001s
I20250116 02:59:47.475713 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.460263 20753 raft_consensus.cc:2405] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250116 02:59:47.483093 20679 raft_consensus.cc:1270] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 02:59:47.413854 20752 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:47.461289 20750 raft_consensus.cc:2371] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6. Re-sending same reply.
I20250116 02:59:47.475593 20720 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:47.446856 23592 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:47.480080 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.481317 20647 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:47.463835 20754 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
W20250116 02:59:47.487084 20572 leader_election.cc:336] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.636s (SENT)
I20250116 02:59:47.491114 20572 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:47.455308 23553 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:47.493948 23553 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:47.497159 23670 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:47.498929 20757 raft_consensus.cc:2405] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250116 02:59:47.500278 20757 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250116 02:59:47.500602 20757 pending_rounds.cc:77] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 6
I20250116 02:59:47.500820 20757 pending_rounds.cc:99] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250116 02:59:47.501032 20757 raft_consensus.cc:2883] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:47.501394 20606 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250116 02:59:47.502033 23598 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:47.447836 23623 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:47.502719 23598 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:47.502969 23623 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:47.478892 23608 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Advancing to term 9
I20250116 02:59:47.505293 23623 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:47.445906 23507 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:47.507076 23507 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:47.502068 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 635) took 1008 ms. Trace:
I20250116 02:59:47.509019 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:47.509286 23507 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:47.508311 20755 rpcz_store.cc:276] 0116 02:59:46.493216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:46.493341 (+ 125us) service_pool.cc:225] Handling call
0116 02:59:46.501849 (+ 8508us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:46.502239 (+ 390us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:46.502283 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:46.524067 (+ 21784us) log.cc:844] Serialized 52 byte log entry
0116 02:59:47.148087 (+624020us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:47.158921 (+ 10834us) log.cc:844] Serialized 14 byte log entry
0116 02:59:47.159179 (+ 258us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:47.159260 (+ 81us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:47.501794 (+342534us) raft_consensus.cc:1680] finished
0116 02:59:47.501816 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:47.501998 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10842,"prepare.queue_time_us":14188,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":25779,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":23140,"spinlock_wait_cycles":10025856,"thread_start_us":376047,"threads_started":1,"wal-append.queue_time_us":458897}
I20250116 02:59:47.425622 23681 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:47.475941 23561 consensus_meta.cc:287] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.151s user 0.012s sys 0.003s
I20250116 02:59:47.503181 23699 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20250116 02:59:47.519464 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 243316us
I20250116 02:59:47.521608 20757 raft_consensus.cc:1270] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250116 02:59:47.524077 20754 pending_rounds.cc:77] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 1
I20250116 02:59:47.524217 20676 raft_consensus.cc:2405] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250116 02:59:47.527333 20754 pending_rounds.cc:99] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250116 02:59:47.527597 20754 raft_consensus.cc:2883] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250116 02:59:47.528177 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 163405us
I20250116 02:59:47.537222 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:47.540189 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 59032us
I20250116 02:59:47.494522 23553 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:47.523176 23632 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250116 02:59:47.593312 21331 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:36880
I20250116 02:59:47.594899 21331 raft_consensus.cc:604] T 8d8e1b8edadd42f08808ffaf11b40c8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:47.532104 23592 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:47.540292 23681 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:47.546463 20748 raft_consensus.cc:1270] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
W20250116 02:59:47.627331 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 67593.4us
I20250116 02:59:47.575487 23507 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:47.578339 23553 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:47.653288 20748 pending_rounds.cc:77] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 2
I20250116 02:59:47.540508 23608 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:47.699815 23553 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:47.702205 20748 pending_rounds.cc:99] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250116 02:59:47.931047 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.937295 23701 raft_consensus.cc:2743] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:47.940979 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 465442us
I20250116 02:59:47.942157 20720 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:47.943707 20720 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:47.945776 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:47.816387 20748 raft_consensus.cc:2883] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250116 02:59:47.944384 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 386331us
I20250116 02:59:47.949633 20680 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
W20250116 02:59:48.033406 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 546292us
I20250116 02:59:47.709415 23507 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:48.072886 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 129241us
I20250116 02:59:48.073997 20720 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:48.040884 20572 leader_election.cc:336] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.686s (SENT)
I20250116 02:59:48.110373 20749 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250116 02:59:48.111726 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 570577us
I20250116 02:59:47.942387 23701 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:48.119194 20750 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.005s sys 0.007s
W20250116 02:59:48.047719 23636 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.021s sys 0.044s
I20250116 02:59:47.941954 23694 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:48.122196 20750 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.595s user 0.006s sys 0.007s
W20250116 02:59:48.140836 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 105602us
W20250116 02:59:48.136727 23636 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.021s sys 0.044s
W20250116 02:59:48.164109 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 123262us
W20250116 02:59:48.167520 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 93641.5us
I20250116 02:59:48.168506 20720 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:48.170712 20755 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.012s sys 0.004s
W20250116 02:59:48.171299 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 48971.6us
W20250116 02:59:48.180884 20755 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s user 0.013s sys 0.004s
W20250116 02:59:48.172044 20572 leader_election.cc:336] T a569f56f9d2143869b29c78321047d00 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.758s (SENT)
W20250116 02:59:48.205458 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.001s sys 0.021s
I20250116 02:59:48.245211 23608 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:48.243635 23686 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:48.256999 20720 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:48.258939 20720 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:48.267100 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 101100us
W20250116 02:59:48.269042 20572 leader_election.cc:336] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.792s (SENT)
I20250116 02:59:48.333148 23632 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:48.347131 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 78127.8us
I20250116 02:59:48.195325 23696 consensus_queue.cc:1035] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.128s
I20250116 02:59:48.382755 23608 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:48.383602 23608 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.386096 23608 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:48.386785 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 188839us
I20250116 02:59:48.385088 23717 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:48.387842 23670 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:48.388795 23622 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:48.387569 20647 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:48.391938 20647 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:48.393290 20647 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:48.394902 23608 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:48.395339 23608 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:48.395771 23608 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:48.248410 23713 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:48.411767 21301 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.003s sys 0.027s
W20250116 02:59:48.414013 21301 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.760s user 0.003s sys 0.027s
W20250116 02:59:48.428470 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 273332us
I20250116 02:59:48.400138 23625 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:48.426535 20754 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.016s sys 0.019s
W20250116 02:59:48.436122 20754 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s user 0.016s sys 0.019s
W20250116 02:59:48.438395 20572 leader_election.cc:336] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.222s (SENT)
I20250116 02:59:48.412992 23623 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:48.464661 23707 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:48.465134 23718 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:48.459770 23507 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:48.463892 23693 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:48.463795 23588 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:48.446484 23368 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:48.464133 23708 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:48.472769 20752 consensus_meta.cc:287] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.984s user 0.011s sys 0.002s
I20250116 02:59:48.469053 23698 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:48.527698 20752 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:48.440932 23625 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.486546 23561 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:48.536825 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.820s user 0.001s sys 0.007s
W20250116 02:59:48.566088 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.862s user 0.001s sys 0.008s
I20250116 02:59:48.572814 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1659) took 1205 ms. Trace:
I20250116 02:59:48.573735 20752 rpcz_store.cc:276] 0116 02:59:47.331651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.400003 (+ 68352us) service_pool.cc:225] Handling call
0116 02:59:48.536625 (+1136622us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:48.469782 23623 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.534231 23696 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:48.641790 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:48.490722 23507 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:48.543404 23728 consensus_queue.cc:1035] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.071s
I20250116 02:59:48.500485 23368 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.334129 23632 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:48.491088 23588 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.391352 23622 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.129272 23701 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.489554 23707 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:48.142119 23694 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.258198 23686 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.424537 23713 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:48.490912 23693 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.388410 23670 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.526592 23708 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:48.642194 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:48.735213 23534 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:49.364136 23707 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:49.364260 23632 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:48.924479 23703 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:49.365672 23703 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:49.367946 23703 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:49.368575 23718 consensus_meta.cc:287] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.882s user 0.006s sys 0.002s
I20250116 02:59:49.369256 23718 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250116 02:59:48.857510 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 598661us
W20250116 02:59:48.689106 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 227307us
I20250116 02:59:48.547775 23698 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:49.372661 23708 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:49.373781 23698 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:49.374215 23623 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:48.992724 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 144777us
I20250116 02:59:49.378126 23368 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:49.361023 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 967701us
I20250116 02:59:49.380913 20647 leader_election.cc:304] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:49.360965 21331 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Advancing to term 9
I20250116 02:59:48.999759 23739 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:48.999121 23732 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:49.391360 20720 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:49.365023 23632 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:49.001411 23710 log.cc:926] Time spent T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.467s user 0.000s sys 0.001s
I20250116 02:59:49.415271 20647 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:49.419549 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:49.419965 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:49.416697 20647 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:49.422065 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:49.364921 23707 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:49.296452 23721 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:49.308096 23729 log.cc:926] Time spent T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.568s user 0.003s sys 0.000s
W20250116 02:59:49.279166 23714 log.cc:926] Time spent T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.580s user 0.000s sys 0.002s
W20250116 02:59:49.382850 23553 consensus_meta.cc:287] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.512s user 0.000s sys 0.015s
W20250116 02:59:49.246778 23715 log.cc:926] Time spent T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.598s user 0.000s sys 0.002s
W20250116 02:59:49.002422 23730 log.cc:926] Time spent T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.172s user 0.000s sys 0.000s
W20250116 02:59:49.000651 23724 log.cc:926] Time spent T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20250116 02:59:49.361996 23507 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:49.245872 23625 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:49.000420 23709 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:49.420794 20604 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
W20250116 02:59:49.435995 20681 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.001s sys 0.013s
W20250116 02:59:48.898583 23608 consensus_meta.cc:287] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.502s user 0.010s sys 0.005s
I20250116 02:59:49.440861 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:49.365051 23588 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:49.470201 20681 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s user 0.002s sys 0.013s
I20250116 02:59:49.441690 23561 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:49.436548 23696 raft_consensus.cc:2743] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:49.481693 20604 raft_consensus.cc:1270] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250116 02:59:49.473049 21331 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 9.
I20250116 02:59:49.527282 23565 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:49.527590 23591 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:49.426940 23632 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:49.529065 23635 log.cc:926] Time spent T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 3.355s user 0.005s sys 0.043s
I20250116 02:59:49.407871 23739 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:49.530683 23622 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:49.531515 23670 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:49.621701 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:49.532557 23507 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:49.482169 23709 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:49.533098 23561 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:49.362213 23534 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:49.438697 23707 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:49.537820 23588 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:49.587828 23714 log.cc:926] Time spent T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20250116 02:59:49.456441 23553 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:49.700750 23689 raft_consensus.cc:988] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:49.695240 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1646) took 2331 ms. Trace:
W20250116 02:59:49.651032 23729 log.cc:926] Time spent T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20250116 02:59:49.713980 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:49.736148 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 300228us
I20250116 02:59:49.717800 20753 rpcz_store.cc:276] 0116 02:59:47.321358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.514376 (+193018us) service_pool.cc:225] Handling call
0116 02:59:47.514759 (+ 383us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:47.716190 (+201431us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:47.716229 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:48.074676 (+358447us) log.cc:844] Serialized 26 byte log entry
0116 02:59:48.158595 (+ 83919us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:48.196675 (+ 38080us) log.cc:844] Serialized 14 byte log entry
0116 02:59:48.498724 (+302049us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:48.498951 (+ 227us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:49.652565 (+1153614us) raft_consensus.cc:1680] finished
0116 02:59:49.652583 (+ 18us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:49.652715 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18644,"prepare.queue_time_us":39070,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":8483,"raft.queue_time_us":185835,"spinlock_wait_cycles":122618624,"thread_start_us":514067,"threads_started":3,"wal-append.queue_time_us":357295}
I20250116 02:59:49.723011 23588 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:49.728194 23632 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:49.448092 23721 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:49.413223 23732 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:49.729950 23754 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:49.730957 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1638) took 2440 ms. Trace:
I20250116 02:59:49.747373 20756 rpcz_store.cc:276] 0116 02:59:47.290187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.291057 (+ 870us) service_pool.cc:225] Handling call
0116 02:59:47.291460 (+ 403us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:47.468594 (+177134us) raft_consensus.cc:1529] Early marking committed up to index 3
0116 02:59:47.513847 (+ 45253us) log.cc:844] Serialized 14 byte log entry
0116 02:59:47.837262 (+323415us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:48.033427 (+196165us) log.cc:844] Serialized 26 byte log entry
0116 02:59:48.033673 (+ 246us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:48.115839 (+ 82166us) log.cc:844] Serialized 14 byte log entry
0116 02:59:48.256827 (+140988us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:48.256915 (+ 88us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:49.730708 (+1473793us) raft_consensus.cc:1680] finished
0116 02:59:49.730732 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:49.730879 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":116031,"prepare.queue_time_us":115157,"prepare.run_cpu_time_us":179,"prepare.run_wall_time_us":8214,"raft.queue_time_us":425257,"raft.run_cpu_time_us":418,"raft.run_wall_time_us":61896,"thread_start_us":518211,"threads_started":3,"wal-append.queue_time_us":954033}
W20250116 02:59:49.731727 23679 log.cc:926] Time spent T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 2.401s user 0.024s sys 0.028s
I20250116 02:59:49.731753 21331 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250116 02:59:49.692780 23670 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:49.732079 23667 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:49.480732 23608 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:49.737433 20649 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:49.736296 21333 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:49.652804 23561 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:49.754532 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 504) took 3917 ms. Trace:
W20250116 02:59:49.732513 23653 log.cc:926] Time spent T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 2.998s user 0.011s sys 0.032s
I20250116 02:59:49.731895 23707 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:49.753917 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 948) took 2240 ms. Trace:
W20250116 02:59:49.760958 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 344314us
I20250116 02:59:49.753365 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 500) took 3918 ms. Trace:
I20250116 02:59:49.765733 21332 rpcz_store.cc:276] 0116 02:59:47.513604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.514010 (+ 406us) service_pool.cc:225] Handling call
0116 02:59:47.514396 (+ 386us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:47.524434 (+ 10038us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:47.526262 (+ 1828us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:47.532764 (+ 6502us) log.cc:844] Serialized 52 byte log entry
0116 02:59:48.040424 (+507660us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:48.040478 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:48.040572 (+ 94us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:49.753668 (+1713096us) raft_consensus.cc:1680] finished
0116 02:59:49.753689 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:49.753839 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":63164,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":13830,"thread_start_us":447778,"threads_started":1,"wal-append.queue_time_us":584455}
W20250116 02:59:49.758570 23730 log.cc:926] Time spent T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250116 02:59:49.728237 23553 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:49.588407 23591 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:49.745194 23507 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:49.818302 20683 raft_consensus.cc:1235] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:49.763068 20600 rpcz_store.cc:276] 0116 02:59:45.836966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:45.860565 (+ 23599us) service_pool.cc:225] Handling call
0116 02:59:45.860864 (+ 299us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:45.902018 (+ 41154us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:45.902053 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:45.934746 (+ 32693us) log.cc:844] Serialized 26 byte log entry
0116 02:59:45.983729 (+ 48983us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:45.983769 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:45.983842 (+ 73us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:49.412482 (+3428640us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:49.735324 (+322842us) raft_consensus.cc:1680] finished
0116 02:59:49.735350 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:49.754466 (+ 19116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9365,"raft.queue_time_us":58492,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":25883,"spinlock_wait_cycles":8292736,"thread_start_us":61316,"threads_started":2,"wal-append.queue_time_us":122898}
I20250116 02:59:50.153966 20679 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 8. Current term is 9. Ops: []
I20250116 02:59:50.112680 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 643) took 2409 ms. Trace:
W20250116 02:59:50.111004 20572 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.246s user 0.014s sys 0.011s
W20250116 02:59:50.160707 20572 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.295s user 0.017s sys 0.015s
I20250116 02:59:49.779611 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:49.768296 20599 rpcz_store.cc:276] 0116 02:59:45.834515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:45.835039 (+ 524us) service_pool.cc:225] Handling call
0116 02:59:45.849403 (+ 14364us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:45.913324 (+ 63921us) raft_consensus.cc:1529] Early marking committed up to index 5
0116 02:59:46.141608 (+228284us) log.cc:844] Serialized 14 byte log entry
0116 02:59:46.877337 (+735729us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b18003c4d80. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32698be4b 00007fc326991913 00007fc326991e2b 00007fc326d1997a
0116 02:59:47.154186 (+276849us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:47.155328 (+ 1142us) log.cc:844] Serialized 26 byte log entry
0116 02:59:47.155479 (+ 151us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:47.158139 (+ 2660us) log.cc:844] Serialized 14 byte log entry
0116 02:59:47.159462 (+ 1323us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:47.159617 (+ 155us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:48.154576 (+994959us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b18003c4d80. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd67e8 00007fc323dd9cc1 00007fc323dd9c79 00007fc323dd9c09 00007fc323dd9bd1 00007fc323dd8ecd 00007fc32ce13234 00007fc32ce13069 00007fc323db8fe6
0116 02:59:49.730918 (+1576342us) spinlock_profiling.cc:243] Waited 5.86 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:49.753071 (+ 22153us) raft_consensus.cc:1680] finished
0116 02:59:49.753097 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:49.753283 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5514,"prepare.queue_time_us":210724,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":89,"raft.queue_time_us":70240,"raft.run_cpu_time_us":521,"raft.run_wall_time_us":26150,"spinlock_wait_cycles":25782144,"thread_start_us":541050,"threads_started":3,"wal-append.queue_time_us":1044698}
I20250116 02:59:49.755808 23589 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:50.160452 20757 rpcz_store.cc:276] 0116 02:59:47.703456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.703660 (+ 204us) service_pool.cc:225] Handling call
0116 02:59:50.111703 (+2408043us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b58001f0598. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:50.111989 (+ 286us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:50.112253 (+ 264us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:50.112283 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:50.112332 (+ 49us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:50.112376 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:50.112441 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:50.112600 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38032128}
I20250116 02:59:49.760676 23671 raft_consensus.cc:1076] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:49.732291 23745 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:49.987061 23751 log.cc:926] Time spent T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
W20250116 02:59:49.837778 23657 log.cc:926] Time spent T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 2.787s user 0.019s sys 0.030s
I20250116 02:59:49.904918 23694 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:50.170527 20572 leader_election.cc:336] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.009s (SENT)
W20250116 02:59:50.171566 20647 connection.cc:232] RPC call timeout handler was delayed by -1.5440327599999364s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2146s for a response
I20250116 02:59:50.180655 20572 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:49.836333 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:49.774178 23689 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:50.112680 23739 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:49.771526 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.113127 23608 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:49.848551 23693 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:49.847023 23753 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:49.970088 23701 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:50.159679 23586 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:50.159706 23549 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:49.839259 23534 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:49.844847 23709 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:49.838140 23721 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:50.223229 23753 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.156730 23686 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:49.801102 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.224436 23591 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:49.910754 23731 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.380s
I20250116 02:59:50.225371 23753 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:49.765430 23670 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:50.201835 23752 log.cc:926] Time spent T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.400s user 0.002s sys 0.000s
I20250116 02:59:50.228175 23670 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:50.229192 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 517) took 1654 ms. Trace:
I20250116 02:59:50.229780 20606 rpcz_store.cc:276] 0116 02:59:48.574597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:48.574798 (+ 201us) service_pool.cc:225] Handling call
0116 02:59:49.365744 (+790946us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:49.366113 (+ 369us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:49.366140 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:49.366665 (+ 525us) log.cc:844] Serialized 26 byte log entry
0116 02:59:49.456954 (+ 90289us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:49.457001 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:49.457094 (+ 93us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:50.228921 (+771827us) raft_consensus.cc:1680] finished
0116 02:59:50.228948 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:50.229113 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2479,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":89824,"threads_started":1,"wal-append.queue_time_us":327272}
I20250116 02:59:50.203943 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1640) took 2890 ms. Trace:
I20250116 02:59:50.231491 20751 rpcz_store.cc:276] 0116 02:59:47.313991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.315367 (+ 1376us) service_pool.cc:225] Handling call
0116 02:59:47.315777 (+ 410us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:47.316203 (+ 426us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:47.316232 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:47.316727 (+ 495us) log.cc:844] Serialized 26 byte log entry
0116 02:59:47.532100 (+215373us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:47.591025 (+ 58925us) log.cc:844] Serialized 14 byte log entry
0116 02:59:49.911734 (+2320709us) spinlock_profiling.cc:243] Waited 8.68 ms on lock 0x7b18001883e0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd3305 00007fc326d073d0 00007fc326d19666 00007fc326d2a1d7 00007fc326d2a158
0116 02:59:50.201994 (+290260us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:50.202106 (+ 112us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:50.202161 (+ 55us) raft_consensus.cc:1680] finished
0116 02:59:50.203699 (+ 1538us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:50.203874 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":407141,"prepare.queue_time_us":56041,"prepare.run_cpu_time_us":149,"prepare.run_wall_time_us":27414,"raft.queue_time_us":362833,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":59088,"spinlock_wait_cycles":19959680,"thread_start_us":477363,"threads_started":2,"wal-append.queue_time_us":837684,"wal-append.run_cpu_time_us":2982,"wal-append.run_wall_time_us":1087509}
I20250116 02:59:49.843137 23746 consensus_queue.cc:1035] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.752s
W20250116 02:59:50.205137 23565 consensus_meta.cc:287] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.631s user 0.005s sys 0.005s
I20250116 02:59:50.234675 23565 raft_consensus.cc:2743] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:50.197497 23762 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:50.235006 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 64513.6us
W20250116 02:59:50.235986 20572 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.746s (SENT) [suppressed 47 similar messages]
I20250116 02:59:50.235626 23762 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.193856 23689 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:50.238308 23689 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:50.206504 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1737) took 2658 ms. Trace:
I20250116 02:59:50.167224 23507 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:50.189333 23732 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:50.173943 23589 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.238782 20681 rpcz_store.cc:276] 0116 02:59:47.548059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.548171 (+ 112us) service_pool.cc:225] Handling call
0116 02:59:47.613730 (+ 65559us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:48.426000 (+812270us) raft_consensus.cc:1529] Early marking committed up to index 2
0116 02:59:48.567853 (+141853us) log.cc:844] Serialized 14 byte log entry
0116 02:59:49.761480 (+1193627us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:49.773000 (+ 11520us) log.cc:844] Serialized 26 byte log entry
0116 02:59:49.773176 (+ 176us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:49.773222 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:49.773294 (+ 72us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:50.206284 (+432990us) raft_consensus.cc:1680] finished
0116 02:59:50.206301 (+ 17us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:50.206424 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":683230,"prepare.queue_time_us":433165,"prepare.run_cpu_time_us":148,"prepare.run_wall_time_us":146,"raft.queue_time_us":663449,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":39517,"thread_start_us":1336349,"threads_started":4,"wal-append.queue_time_us":1174784}
I20250116 02:59:49.759788 23754 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.242180 23507 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:50.181726 23745 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.242761 23754 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:50.212975 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.218580 23588 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:50.219223 21328 raft_consensus.cc:2388] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:50.198566 23739 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:50.191900 23713 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:50.219880 23765 raft_consensus.cc:2743] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:50.228988 23586 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:50.211607 23608 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:50.244498 23745 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:49.771526 23667 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:50.261639 21299 connection.cc:232] RPC call timeout handler was delayed by -2.438816923000104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
W20250116 02:59:50.262584 20647 leader_election.cc:336] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.796s (SENT)
I20250116 02:59:50.265520 20647 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:50.270208 20647 leader_election.cc:336] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.799s (SENT)
I20250116 02:59:50.270802 20647 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:50.365306 20681 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a569f56f9d2143869b29c78321047d00"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:50.367110 20681 raft_consensus.cc:604] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:50.367560 20681 raft_consensus.cc:612] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:50.371068 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1739) took 2797 ms. Trace:
I20250116 02:59:50.371565 20677 rpcz_store.cc:276] 0116 02:59:47.573536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.573706 (+ 170us) service_pool.cc:225] Handling call
0116 02:59:48.072204 (+498498us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:48.155063 (+ 82859us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:48.155095 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:48.164883 (+ 9788us) log.cc:844] Serialized 52 byte log entry
0116 02:59:49.710240 (+1545357us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b1800286060. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323db8565 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32698be4b 00007fc326991913 00007fc326991cc1 00007fc326ce345f
0116 02:59:49.877751 (+167511us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:50.173357 (+295606us) log.cc:844] Serialized 14 byte log entry
0116 02:59:50.363852 (+190495us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:50.363936 (+ 84us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:50.363982 (+ 46us) raft_consensus.cc:1680] finished
0116 02:59:50.370761 (+ 6779us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:50.370993 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":161521,"prepare.queue_time_us":391965,"raft.queue_time_us":137238,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":4691,"spinlock_wait_cycles":2639232,"thread_start_us":554190,"threads_started":3,"wal-append.queue_time_us":2025322}
I20250116 02:59:50.256479 23762 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:50.376754 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 140872us
W20250116 02:59:50.403379 20572 connection.cc:232] RPC call timeout handler was delayed by -1.1196308779999526s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3547s for a response
W20250116 02:59:50.403987 20572 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.087s (SENT)
I20250116 02:59:50.269408 23756 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:50.326192 23591 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:50.336102 23731 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:50.424369 23586 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:50.428472 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 254992us
I20250116 02:59:50.365501 23736 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:50.431830 20646 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:50.426685 23667 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:50.260056 23739 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:50.435896 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 163903us
I20250116 02:59:50.369956 23732 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:50.262691 23561 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.273572 23608 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:50.445611 20572 leader_election.cc:336] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.152s (SENT)
W20250116 02:59:50.435696 23711 log.cc:926] Time spent T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 1.978s user 0.002s sys 0.011s
I20250116 02:59:50.447227 23732 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.456084 23732 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:50.458155 23561 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:50.460253 23739 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:50.461640 23589 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:50.462466 23608 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:50.438886 23736 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.434232 23756 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.466298 23765 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:50.466971 23765 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:50.467505 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.469285 21329 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:50.469779 23756 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:50.470037 23765 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:50.471320 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:50.446913 23670 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Leader election won for term 9
W20250116 02:59:50.473744 20572 leader_election.cc:336] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.219s (SENT)
I20250116 02:59:50.474256 20572 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:50.475419 20572 leader_election.cc:336] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.289s (SENT)
I20250116 02:59:50.475837 20572 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:50.479885 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:50.482246 21331 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250116 02:59:50.472991 23598 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:50.484393 23598 raft_consensus.cc:2743] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:50.486505 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:50.488278 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.489584 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.492491 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:50.491317 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 82665.6us
I20250116 02:59:50.490262 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:50.469969 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.495429 20681 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:50.492918 21334 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 02:59:50.497659 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.485033 23507 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:50.515563 20571 connection.cc:232] RPC call timeout handler was delayed by -1.5176346849999618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
I20250116 02:59:50.517701 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.518534 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.521809 20608 raft_consensus.cc:2388] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 6: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 02:59:50.522126 20571 connection.cc:232] RPC call timeout handler was delayed by -1.5176346849999618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
W20250116 02:59:50.522508 20571 connection.cc:232] RPC call timeout handler was delayed by -1.494234684999924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15910000000000002s for a response
W20250116 02:59:50.523332 20571 connection.cc:232] RPC call timeout handler was delayed by -1.3547346849999258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
I20250116 02:59:50.523586 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.523883 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.524431 20603 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:50.524490 20600 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:50.526427 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.527175 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.527925 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.530357 23589 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:50.531036 23589 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.527834 20600 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:50.537212 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:50.520009 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.538414 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:50.957748 20678 raft_consensus.cc:2405] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250116 02:59:50.584777 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.959335 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:50.553349 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 73381.8us
I20250116 02:59:50.538414 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.962473 20646 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:50.962718 20608 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:50.555779 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.577840 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.964501 20601 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:50.527221 20571 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:50.966651 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 439473us
I20250116 02:59:50.967567 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:50.668769 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:50.971555 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:50.522953 23670 raft_consensus.cc:695] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [term 9 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:50.953610 23549 consensus_meta.cc:287] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.731s user 0.012s sys 0.000s
I20250116 02:59:50.975718 23561 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:50.998065 23561 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.007710 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.998242 23670 consensus_queue.cc:237] T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:50.975785 23507 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:50.979760 23558 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:50.989725 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.001446 23598 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 02:59:50.994145 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.992678 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.996001 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:50.993254 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.018100 21333 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:50.985412 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:50.983402 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.004626 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:50.705206 20600 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43854
I20250116 02:59:50.533428 23589 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.009032 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:51.001688 23678 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:51.020531 20600 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:59:51.024199 20600 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:51.008540 20604 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 02:59:50.553167 23559 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:50.940601 23586 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:50.940438 23731 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:50.995476 23689 consensus_meta.cc:287] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.757s user 0.013s sys 0.000s
I20250116 02:59:51.032923 23689 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.027470 23678 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:51.035637 20681 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:51.036343 20677 raft_consensus.cc:2363] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7.
I20250116 02:59:51.021673 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.028230 20646 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.014461 23558 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:51.027211 23559 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:51.042723 20676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 560) took 4719 ms (client timeout 1635 ms). Trace:
I20250116 02:59:51.042560 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:51.043263 20676 rpcz_store.cc:269] 0116 02:59:46.323525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:46.323645 (+ 120us) service_pool.cc:225] Handling call
0116 02:59:51.042426 (+4718781us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000870d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:51.042572 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2757740416}
I20250116 02:59:51.043915 23559 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:50.976125 23776 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:51.045925 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.046072 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.037787 23507 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:51.035784 23689 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.045166 23558 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:51.072538 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.058898 23776 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:51.077883 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.080065 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.037395 23586 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.064240 23772 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250116 02:59:51.081480 20646 consensus_peers.cc:487] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb -> Peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959): Couldn't send request to peer 57fcfcd5ab01438c9a4baf8a61871aa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59601d2a275b4efdbbc89dbcb608871c. This is attempt 1: this message will repeat every 5th retry.
I20250116 02:59:51.083396 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.050438 23731 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:51.093406 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.095016 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:51.082541 23776 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:51.096318 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.097610 21336 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:51.105749 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.107563 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.106240 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.110724 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.111151 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.108340 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.112043 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.112579 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:51.113270 21334 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 7
I20250116 02:59:51.114069 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.115638 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.116122 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.120352 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.117928 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.123090 20646 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.096207 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.104395 23736 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.125128 23338 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250116 02:59:51.146554 23786 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20250116 02:59:51.157079 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.125319 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.162750 20600 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.172688 20600 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.165313 21334 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:51.186686 20646 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:51.197412 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.232434 20678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
W20250116 02:59:51.232757 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 128538us
I20250116 02:59:51.233242 20678 raft_consensus.cc:604] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:51.234880 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 48366.3us
I20250116 02:59:51.233621 20678 raft_consensus.cc:612] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250116 02:59:51.233697 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 209050us
I20250116 02:59:51.239454 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2274) took 4067 ms. Trace:
I20250116 02:59:51.238420 21302 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:51.251016 23704 log.cc:926] Time spent T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 2.833s user 0.018s sys 0.015s
I20250116 02:59:51.255645 23586 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.257138 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.257553 23731 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:51.169621 23561 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.260944 23561 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:51.261334 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.260429 23735 raft_consensus.cc:2949] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:59:51.261507 23561 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:51.266868 23788 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:51.267526 23788 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:51.267872 23788 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:51.271811 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2281) took 1846 ms. Trace:
I20250116 02:59:51.272274 20605 rpcz_store.cc:276] 0116 02:59:49.425207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:49.425358 (+ 151us) service_pool.cc:225] Handling call
0116 02:59:49.425678 (+ 320us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:49.534688 (+109010us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:49.534724 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:49.534766 (+ 42us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:49.534796 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:51.271551 (+1736755us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540014d7d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:51.271597 (+ 46us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:51.271746 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":250155264}
I20250116 02:59:51.248452 20602 rpcz_store.cc:276] 0116 02:59:47.172080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.172220 (+ 140us) service_pool.cc:225] Handling call
0116 02:59:47.184297 (+ 12077us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:47.212181 (+ 27884us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:51.239177 (+4026996us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b54002e59d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:51.239388 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61751680}
I20250116 02:59:51.275832 23507 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250116 02:59:51.235991 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 135656us
W20250116 02:59:51.280342 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 168915us
W20250116 02:59:51.282786 20572 leader_election.cc:336] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.832s (SENT)
I20250116 02:59:51.284849 20572 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:51.286929 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:51.284010 20600 raft_consensus.cc:2949] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:59:51.238765 23338 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election won for term 7
W20250116 02:59:51.293325 20572 leader_election.cc:336] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.840s (SENT)
W20250116 02:59:51.285670 23785 log.cc:926] Time spent T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
W20250116 02:59:51.296191 20572 leader_election.cc:336] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.872s (SENT)
I20250116 02:59:51.300253 23490 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:51.299964 23565 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.005s
I20250116 02:59:51.301338 21302 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:51.302500 20572 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:51.307803 23598 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:51.312979 20678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
W20250116 02:59:51.311695 20572 leader_election.cc:336] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 3.547s (SENT)
I20250116 02:59:51.323860 20678 raft_consensus.cc:604] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.328676 21299 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:51.336014 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 942) took 4142 ms. Trace:
I20250116 02:59:51.336196 21302 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.343575 21302 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.346006 21302 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:51.310148 23490 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:51.312419 23507 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250116 02:59:51.347590 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 55219.9us
I20250116 02:59:51.336556 21335 rpcz_store.cc:276] 0116 02:59:47.193972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.204883 (+ 10911us) service_pool.cc:225] Handling call
0116 02:59:47.215227 (+ 10344us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:47.215606 (+ 379us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:47.215635 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:47.219600 (+ 3965us) log.cc:844] Serialized 26 byte log entry
0116 02:59:47.678692 (+459092us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:47.678739 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:47.678967 (+ 228us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:51.250336 (+3571369us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:51.328522 (+ 78186us) raft_consensus.cc:1680] finished
0116 02:59:51.328546 (+ 24us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:51.335933 (+ 7387us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":97607,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":11771264,"thread_start_us":458622,"threads_started":1,"wal-append.queue_time_us":1031179}
I20250116 02:59:51.317344 23338 raft_consensus.cc:695] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:51.357628 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 45970.9us
I20250116 02:59:51.334510 23697 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:51.325400 23586 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:51.348027 23793 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:51.351390 23682 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:51.348098 23606 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:51.367925 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.354636 23490 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:51.361725 20572 leader_election.cc:336] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.014s (SENT)
I20250116 02:59:51.369997 23338 consensus_queue.cc:237] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.362629 23697 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:51.387723 23561 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:51.351650 23699 consensus_queue.cc:1046] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20250116 02:59:51.381867 23793 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:51.393937 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 47015.4us
W20250116 02:59:51.395107 20572 leader_election.cc:336] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.087s (SENT)
I20250116 02:59:51.388200 23532 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 02:59:51.401737 20608 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:51.365311 23801 consensus_queue.cc:1046] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20250116 02:59:51.394716 23697 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
W20250116 02:59:51.401635 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 55721.7us
I20250116 02:59:51.403656 20572 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:51.414909 21302 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:51.411397 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 47346us
I20250116 02:59:51.380774 23586 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:51.403704 23793 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:51.418723 20572 leader_election.cc:336] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.411s (SENT)
I20250116 02:59:51.409065 23532 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:51.423938 23636 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250116 02:59:51.423173 20572 leader_election.cc:336] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.758s (SENT)
I20250116 02:59:51.421940 23699 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 02:59:51.436625 21302 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.418207 23712 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:51.439286 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1644) took 4120 ms. Trace:
I20250116 02:59:51.427624 23722 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:51.435304 23532 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Advancing to term 9
I20250116 02:59:51.434309 23723 raft_consensus.cc:2743] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:51.420514 23731 consensus_queue.cc:1046] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20250116 02:59:51.442723 20750 rpcz_store.cc:276] 0116 02:59:47.320815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.515082 (+194267us) service_pool.cc:225] Handling call
0116 02:59:47.515404 (+ 322us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:48.150818 (+635414us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:48.150870 (+ 52us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:48.165319 (+ 14449us) log.cc:844] Serialized 26 byte log entry
0116 02:59:48.442022 (+276703us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:48.487575 (+ 45553us) log.cc:844] Serialized 14 byte log entry
0116 02:59:51.438751 (+2951176us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:51.438969 (+ 218us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:51.439033 (+ 64us) raft_consensus.cc:1680] finished
0116 02:59:51.439045 (+ 12us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:51.439215 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2980285,"prepare.queue_time_us":45675,"prepare.run_cpu_time_us":16705,"prepare.run_wall_time_us":319769,"raft.queue_time_us":250660,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":69201,"thread_start_us":851273,"threads_started":2,"wal-append.queue_time_us":371151,"wal-append.run_cpu_time_us":22123,"wal-append.run_wall_time_us":2628482}
I20250116 02:59:51.440447 23613 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250116 02:59:51.426028 23558 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.461621 23763 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:51.462128 23699 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:51.453521 23734 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:51.501605 23800 log.cc:926] Time spent T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
W20250116 02:59:51.511155 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 2.93552e+06us
W20250116 02:59:51.513300 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 2.12178e+06us
I20250116 02:59:51.475602 23718 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:51.475672 23607 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:51.555310 20604 raft_consensus.cc:2388] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:51.460870 23738 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:51.560905 20720 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.578281 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.516978 23713 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:51.459450 23712 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:51.516513 23701 consensus_queue.cc:1035] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
W20250116 02:59:51.663097 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 102303us
I20250116 02:59:51.664083 20720 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.555099 23763 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.481285 23722 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.584761 23613 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:51.564006 23718 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.563786 23607 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.699021 20720 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.704313 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:51.708717 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:51.711856 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:51.715950 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:51.718005 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:51.718863 20752 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:51.566283 23738 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.574195 23804 consensus_queue.cc:1046] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.120s
I20250116 02:59:51.678247 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:51.677723 23586 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:51.692967 23613 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:51.677345 23734 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:51.671700 23490 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.750998 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.834914 20607 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:51.765431 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2297) took 1210 ms. Trace:
I20250116 02:59:51.841181 20606 rpcz_store.cc:276] 0116 02:59:50.555489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:50.964676 (+409187us) service_pool.cc:225] Handling call
0116 02:59:51.025094 (+ 60418us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:51.025342 (+ 248us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:51.025371 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:51.025424 (+ 53us) raft_consensus.cc:1640] Marking committed up to 8
0116 02:59:51.025458 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:51.025518 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:51.765213 (+739695us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b54002e6dd0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:51.765351 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34594176}
I20250116 02:59:51.827986 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:51.832401 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:51.828109 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.733484 20720 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:51.847059 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 113685us
I20250116 02:59:51.746019 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.847972 20720 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.848160 20602 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:51.849313 20720 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.850777 20720 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.852046 20720 leader_election.cc:304] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.692917 23601 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:51.701376 23611 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:51.697731 23713 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:51.762717 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 300512us
I20250116 02:59:51.750895 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:51.928797 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 76838.3us
W20250116 02:59:51.744220 23809 log.cc:926] Time spent T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20250116 02:59:51.831002 20753 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:51.733098 23741 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:51.740377 23716 log.cc:926] Time spent T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 3.276s user 0.009s sys 0.027s
I20250116 02:59:51.941941 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:51.935819 20720 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.738080 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 954) took 1307 ms. Trace:
I20250116 02:59:51.762398 23763 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:51.929199 23776 consensus_meta.cc:287] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.815s user 0.008s sys 0.006s
I20250116 02:59:51.737985 23607 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.966331 23738 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.937567 23723 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:51.968915 23712 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:51.969277 23678 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:51.970980 23776 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.973359 23776 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.748795 23718 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.749884 23613 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:51.943908 20720 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:51.974936 20678 raft_consensus.cc:2363] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8.
I20250116 02:59:51.975543 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 588) took 2324 ms. Trace:
I20250116 02:59:51.831063 23490 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:51.981504 20608 raft_consensus.cc:3049] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 8 LEADER]: Stepping down as leader of term 8
I20250116 02:59:51.988615 20608 raft_consensus.cc:738] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:51.988829 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:51.989854 20680 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:51.742522 23807 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:51.929864 23808 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250116 02:59:51.964850 23787 log.cc:926] Time spent T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.686s user 0.000s sys 0.004s
I20250116 02:59:51.940025 23713 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.961056 23812 raft_consensus.cc:3049] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:51.984249 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:52.024293 23812 raft_consensus.cc:738] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:52.024554 20601 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:52.025501 23812 consensus_queue.cc:260] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:51.977062 21327 rpcz_store.cc:276] 0116 02:59:49.651077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:49.652316 (+ 1239us) service_pool.cc:225] Handling call
0116 02:59:49.726821 (+ 74505us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:50.243767 (+516946us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:50.243803 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:50.244390 (+ 587us) log.cc:844] Serialized 26 byte log entry
0116 02:59:50.415228 (+170838us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:50.418468 (+ 3240us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:51.975067 (+1556599us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b54004a06d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:51.975130 (+ 63us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:51.975196 (+ 66us) raft_consensus.cc:1680] finished
0116 02:59:51.975206 (+ 10us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:51.975473 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70300,"raft.queue_time_us":7484,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7333,"spinlock_wait_cycles":1187653888,"thread_start_us":128811,"threads_started":1,"wal-append.queue_time_us":194703,"wal-append.run_cpu_time_us":2050,"wal-append.run_wall_time_us":1018319}
I20250116 02:59:52.027026 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 528) took 1499 ms. Trace:
I20250116 02:59:52.027432 20603 rpcz_store.cc:276] 0116 02:59:50.527816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:50.527941 (+ 125us) service_pool.cc:225] Handling call
0116 02:59:50.528302 (+ 361us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:50.538581 (+ 10279us) raft_consensus.cc:1529] Early marking committed up to index 6
0116 02:59:50.538601 (+ 20us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:50.678599 (+139998us) log.cc:844] Serialized 26 byte log entry
0116 02:59:51.093152 (+414553us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:51.093193 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:51.093282 (+ 89us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:51.774435 (+681153us) spinlock_profiling.cc:243] Waited 8.24 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:52.026696 (+252261us) raft_consensus.cc:1680] finished
0116 02:59:52.026716 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:52.026959 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":130,"spinlock_wait_cycles":18940672,"thread_start_us":315488,"threads_started":1,"wal-append.queue_time_us":523994}
I20250116 02:59:52.034368 20601 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:52.036046 20649 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:52.037098 23581 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election won for term 8
I20250116 02:59:52.037658 23581 raft_consensus.cc:695] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:52.038187 23736 raft_consensus.cc:3049] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:52.038661 23736 raft_consensus.cc:738] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:52.038688 23581 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:52.043668 20678 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:51.947907 21328 rpcz_store.cc:276] 0116 02:59:50.430761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:50.432213 (+ 1452us) service_pool.cc:225] Handling call
0116 02:59:51.661859 (+1229646us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:51.737982 (+ 76123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53109632}
I20250116 02:59:51.997129 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1669) took 4658 ms. Trace:
W20250116 02:59:51.997602 23532 consensus_meta.cc:287] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.518s user 0.011s sys 0.000s
I20250116 02:59:51.997998 23722 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.952150 23805 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:51.937650 23601 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:51.938102 23734 raft_consensus.cc:2743] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:51.937287 23611 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.927599 23810 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:51.933907 23701 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:51.938393 23558 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:51.745401 23788 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:51.990197 20608 consensus_queue.cc:260] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:52.119918 20608 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Advancing to term 9
I20250116 02:59:52.019618 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:52.121264 20602 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:52.053634 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:52.055637 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:52.057814 20681 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:52.059911 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:52.060443 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.006s sys 0.006s
I20250116 02:59:52.123750 20649 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:52.124030 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.618s user 0.007s sys 0.007s
W20250116 02:59:52.124380 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 618389us
W20250116 02:59:52.047439 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.007s sys 0.005s
I20250116 02:59:52.054850 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:52.062325 20755 rpcz_store.cc:276] 0116 02:59:47.338736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.511422 (+172686us) service_pool.cc:225] Handling call
0116 02:59:47.511694 (+ 272us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:47.523847 (+ 12153us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:47.523880 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:47.526484 (+ 2604us) log.cc:844] Serialized 26 byte log entry
0116 02:59:48.199905 (+673421us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:48.269929 (+ 70024us) log.cc:844] Serialized 14 byte log entry
0116 02:59:51.410971 (+3141042us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b1800343d80. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323d4e03a 00007fc32d20abe9 00007fc323c0f930 00007fc323dd3a14 00007fc323dd3305 00007fc326d073d0 00007fc326d19666 00007fc326d2a1d7 00007fc326d2a158
0116 02:59:51.435597 (+ 24626us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:51.435694 (+ 97us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:51.739975 (+304281us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b2400134fd8. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323c2d9d4 00007fc323c2badb 00007fc323cba289 00007fc323cb9f15 00007fc323cb955d 00007fc326975d3e 00007fc32698d4cc 00007fc32698c7f6 00007fc32698c418 00007fc326995df1
0116 02:59:51.996116 (+256141us) raft_consensus.cc:1680] finished
0116 02:59:51.996144 (+ 28us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:51.996996 (+ 852us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":166936,"prepare.queue_time_us":124720,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":3800,"raft.queue_time_us":3375480,"raft.run_cpu_time_us":455,"raft.run_wall_time_us":80035,"spinlock_wait_cycles":58098304,"thread_start_us":649816,"threads_started":2,"wal-append.queue_time_us":899581}
I20250116 02:59:52.028048 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
W20250116 02:59:52.083432 21335 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.001s sys 0.014s
W20250116 02:59:52.060359 23682 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.001s sys 0.012s
W20250116 02:59:52.098271 20647 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.002s sys 0.013s
W20250116 02:59:52.135829 20647 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.571s user 0.002s sys 0.013s
W20250116 02:59:52.136161 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 572091us
I20250116 02:59:52.046047 23613 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250116 02:59:52.100991 21301 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.012s sys 0.004s
W20250116 02:59:52.142781 21301 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.603s user 0.012s sys 0.004s
I20250116 02:59:52.146143 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:52.147431 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:52.149924 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:52.150185 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:52.151932 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:52.124639 20754 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.000s sys 0.019s
W20250116 02:59:52.126818 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.672s user 0.007s sys 0.005s
W20250116 02:59:52.155182 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 700179us
W20250116 02:59:52.127337 21335 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.674s user 0.002s sys 0.014s
W20250116 02:59:52.157696 20754 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.724s user 0.000s sys 0.019s
W20250116 02:59:52.159075 20720 connection.cc:232] RPC call timeout handler was delayed by -3.2229574929999671s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
I20250116 02:59:52.162693 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.163884 20752 raft_consensus.cc:2405] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250116 02:59:52.164943 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.165683 20752 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:52.167029 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.169134 21327 raft_consensus.cc:1235] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 6. Current term is 7. Ops: []
W20250116 02:59:52.135533 23682 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s user 0.001s sys 0.013s
I20250116 02:59:52.137593 21300 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:52.138190 20749 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:52.141922 20680 raft_consensus.cc:2388] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:52.172813 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.174110 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.118414 23808 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:52.159178 23717 raft_consensus.cc:2743] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:52.082408 23532 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:52.154371 23580 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:52.181905 21299 leader_election.cc:400] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8.
I20250116 02:59:52.222298 21299 leader_election.cc:403] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20250116 02:59:52.184441 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 46926.3us
I20250116 02:59:52.179330 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1645) took 4837 ms. Trace:
W20250116 02:59:52.184266 23793 consensus_meta.cc:287] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.752s user 0.007s sys 0.005s
W20250116 02:59:52.082152 23507 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.005s sys 0.008s
I20250116 02:59:52.218686 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:52.245702 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 418138us
I20250116 02:59:52.083940 23736 consensus_queue.cc:260] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:52.361411 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.201771 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:52.357543 20723 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:52.207871 23616 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250116 02:59:52.358414 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 134354us
I20250116 02:59:52.375265 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:52.376196 20678 raft_consensus.cc:1235] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:52.377303 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 02:59:52.232197 23670 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.977s user 0.000s sys 0.024s
I20250116 02:59:52.171264 23805 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:52.176568 23735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:41210 (request call id 17) took 5985 ms. Trace:
W20250116 02:59:52.359177 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 135529us
I20250116 02:59:52.362946 20755 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:52.369534 20723 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:52.224598 20754 rpcz_store.cc:276] 0116 02:59:47.321109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:47.511961 (+190852us) service_pool.cc:225] Handling call
0116 02:59:47.512557 (+ 596us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:47.623223 (+110666us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:47.623258 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:47.628123 (+ 4865us) log.cc:844] Serialized 26 byte log entry
0116 02:59:48.441342 (+813219us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:48.460927 (+ 19585us) log.cc:844] Serialized 14 byte log entry
0116 02:59:52.158220 (+3697293us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:52.158320 (+ 100us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:52.158374 (+ 54us) raft_consensus.cc:1680] finished
0116 02:59:52.158387 (+ 13us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:52.158674 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3140442,"prepare.queue_time_us":44920,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":26503,"raft.queue_time_us":3040871,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":6042,"thread_start_us":1392479,"threads_started":2,"wal-append.queue_time_us":860457,"wal-append.run_cpu_time_us":48326,"wal-append.run_wall_time_us":2635258}
I20250116 02:59:52.371170 21331 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
W20250116 02:59:52.358520 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 69349.6us
I20250116 02:59:52.222147 23580 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:52.231518 23801 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.006s sys 0.017s
W20250116 02:59:52.355854 23704 log.cc:926] Time spent T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20250116 02:59:52.403946 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.439170 20752 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:52.440369 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.432636 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:52.419265 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:52.402827 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.426882 20755 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
W20250116 02:59:52.467788 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 63942.5us
W20250116 02:59:52.470379 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 69515.3us
I20250116 02:59:52.473359 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.474339 20752 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:52.230468 23338 consensus_queue.cc:1035] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.182s
I20250116 02:59:52.479349 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.480237 20755 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:52.469368 23827 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20250116 02:59:52.481243 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.481637 23827 consensus_queue.cc:1035] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250116 02:59:52.482329 23611 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:52.479315 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.483860 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.396068 23558 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.088s
W20250116 02:59:52.484359 23826 log.cc:926] Time spent T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250116 02:59:52.484525 20754 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:52.485481 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:52.470422 23722 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20250116 02:59:52.208353 23808 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:52.493638 23808 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:52.411490 23626 consensus_queue.cc:1035] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20250116 02:59:52.482489 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.504403 20755 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:52.505563 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.381742 23824 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:52.509970 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:52.506312 20755 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:52.599936 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:52.393810 23670 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.133s user 0.000s sys 0.025s
I20250116 02:59:52.589218 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.615792 20752 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
W20250116 02:59:52.422590 23801 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.849s user 0.006s sys 0.017s
W20250116 02:59:52.378459 23507 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.874s user 0.018s sys 0.019s
W20250116 02:59:52.608074 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 139422us
I20250116 02:59:52.619971 20755 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:52.275766 23793 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:52.421810 23580 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:52.623430 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:52.437858 23810 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:52.634900 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.386999 23616 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:52.667589 20752 raft_consensus.cc:2463] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:52.481678 23532 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:52.672845 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.675364 20752 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:52.743698 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.679335 23616 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:52.400393 23735 rpcz_store.cc:276] 0116 02:59:46.175369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:46.179959 (+ 4590us) service_pool.cc:225] Handling call
0116 02:59:46.239011 (+ 59052us) log.cc:844] Serialized 513 byte log entry
0116 02:59:46.876278 (+637267us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b54003e1aa0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326ca0caf 00007fc326c897a5 00007fc326c8e9d1 00007fc326c8e959 00007fc326c8e8e9 00007fc326c8e8b1 00007fc326c8da7d 00007fc32ce13234 00007fc32ce13069 00007fc323dd6866
0116 02:59:49.369831 (+2493553us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b180025eca0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323da3ec6 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326c9d9d1 00007fc326ca8480 00007fc326ca8428
0116 02:59:49.373620 (+ 3789us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b54003e1aa0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326ca0caf 00007fc326c897a5 00007fc326c8e9d1 00007fc326c8e959 00007fc326c8e8e9 00007fc326c8e8b1 00007fc326c8da7d 00007fc32ce13234 00007fc32ce13069 00007fc323dd6866
0116 02:59:50.515752 (+1142132us) spinlock_profiling.cc:243] Waited 7.01 ms on lock 0x7b54003e1aa0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326ca8676 00007fc326ca200a 00007fc326ca1274 00007fc326c897a5 00007fc326c8e9d1 00007fc326c8e959 00007fc326c8e8e9 00007fc326c8e8b1 00007fc326c8da7d 00007fc32ce13234
0116 02:59:50.586631 (+ 70879us) spinlock_profiling.cc:243] Waited 8.18 ms on lock 0x7b54003e1aa0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326c9d9d1 00007fc326ca8480 00007fc326ca8428 00007fc326ca83a8 00007fc326ca8363 00007fc326ca75ec 00007fc32cadfbb6 00007fc32cada568 00007fc326ce421b 00007fc326ce6d10
0116 02:59:52.154310 (+1567679us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58787,"raft.queue_time_us":2950,"raft.run_cpu_time_us":149532,"raft.run_wall_time_us":7514909,"spinlock_wait_cycles":49409024,"thread_start_us":181065,"threads_started":1,"wal-append.queue_time_us":185938,"wal-append.run_cpu_time_us":90826,"wal-append.run_wall_time_us":5219996}
I20250116 02:59:52.631050 23540 consensus_queue.cc:1046] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.139s
I20250116 02:59:52.772119 23788 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:52.772840 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 649) took 1091 ms. Trace:
I20250116 02:59:52.773257 23559 consensus_queue.cc:1046] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250116 02:59:52.774809 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 179175us
I20250116 02:59:52.775086 23559 raft_consensus.cc:3049] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:52.775588 23559 raft_consensus.cc:738] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 02:59:52.776872 20677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 580) took 2295 ms (client timeout 1941 ms). Trace:
I20250116 02:59:52.776469 23559 consensus_queue.cc:260] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:52.777293 20677 rpcz_store.cc:269] 0116 02:59:50.481084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:50.481230 (+ 146us) service_pool.cc:225] Handling call
0116 02:59:52.776610 (+2295380us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b5400325354. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:52.776763 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1247220480}
I20250116 02:59:52.780571 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.773231 20757 rpcz_store.cc:276] 0116 02:59:51.681283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.692347 (+ 11064us) service_pool.cc:225] Handling call
0116 02:59:51.692799 (+ 452us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:51.744781 (+ 51982us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:51.744819 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:51.744871 (+ 52us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:51.744911 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:52.772578 (+1027667us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b5400185954. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:52.772627 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:52.772785 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":118867200}
I20250116 02:59:52.781512 20752 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:52.791554 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:52.412923 23816 consensus_queue.cc:1046] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.282s
I20250116 02:59:52.665720 23689 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:52.765166 23601 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:52.666383 23810 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:52.808138 23559 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:52.811293 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:52.812150 21334 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:52.812256 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:52.813918 21331 raft_consensus.cc:2388] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:52.815460 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:52.817242 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:52.818122 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:52.763537 20647 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:52.818498 21331 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250116 02:59:52.808874 23810 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:52.826341 20678 raft_consensus.cc:2388] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:52.819041 23822 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250116 02:59:52.819535 23806 log.cc:926] Time spent T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.207s user 0.001s sys 0.000s
W20250116 02:59:52.826717 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 63299.2us
W20250116 02:59:52.820649 23828 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.214s user 0.000s sys 0.000s
W20250116 02:59:52.823534 23831 log.cc:926] Time spent T f69213cf66934f1a9aa19c9defd3a4d8 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20250116 02:59:52.827804 23760 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:52.828286 23760 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 02:59:52.825516 23697 consensus_meta.cc:287] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.405s user 0.014s sys 0.000s
I20250116 02:59:52.828778 23760 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:52.829370 23697 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:52.832728 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:52.828187 23670 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 02:59:52.839488 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:52.842254 20647 leader_election.cc:336] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.146s (SENT)
I20250116 02:59:52.842820 20647 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:52.844771 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.841625 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.856896 20647 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:52.845448 23700 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:52.411974 23805 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:52.860402 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.865581 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.873601 20752 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:52.875674 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.786156 23605 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:52.880597 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 666) took 1165 ms. Trace:
I20250116 02:59:52.924449 20751 rpcz_store.cc:276] 0116 02:59:51.714815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.754161 (+ 39346us) service_pool.cc:225] Handling call
0116 02:59:51.754432 (+ 271us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:52.086698 (+332266us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:52.086733 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:52.086783 (+ 50us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:52.086814 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:52.880215 (+793401us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b540007d0d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:52.880295 (+ 80us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:52.880508 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":763539328}
I20250116 02:59:52.926200 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:52.856814 23829 log.cc:926] Time spent T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.370s user 0.000s sys 0.002s
I20250116 02:59:52.899243 21334 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:52.930735 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:52.921092 23601 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:52.932446 23601 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:52.936158 23601 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:52.941505 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1759) took 1177 ms. Trace:
I20250116 02:59:52.942039 20600 rpcz_store.cc:276] 0116 02:59:51.763728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.763967 (+ 239us) service_pool.cc:225] Handling call
0116 02:59:51.764416 (+ 449us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:51.764791 (+ 375us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:51.764819 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:51.765362 (+ 543us) log.cc:844] Serialized 26 byte log entry
0116 02:59:52.086116 (+320754us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:52.086162 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:52.086267 (+ 105us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:52.941266 (+854999us) raft_consensus.cc:1680] finished
0116 02:59:52.941292 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:52.941433 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":265421,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"thread_start_us":320287,"threads_started":1,"wal-append.queue_time_us":720414}
I20250116 02:59:52.874382 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:52.945629 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:52.954169 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.956094 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:52.961421 21328 raft_consensus.cc:1235] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250116 02:59:52.667579 23713 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:52.902438 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.907733 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:52.914707 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:52.677244 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:52.922950 20674 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.081s sys 0.108s
I20250116 02:59:52.884471 23793 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:52.989678 21334 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 7.
I20250116 02:59:52.996045 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:52.900686 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 46282.7us
I20250116 02:59:52.962518 23682 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:52.989696 20749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 655) took 1284 ms (client timeout 1502 ms). Trace:
I20250116 02:59:52.868484 23792 raft_consensus.cc:3049] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Stepping down as leader of term 3
W20250116 02:59:52.992586 20674 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.616s user 0.081s sys 0.108s
I20250116 02:59:53.008599 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.993423 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:52.850487 23689 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:53.012535 23682 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:53.004967 20749 rpcz_store.cc:269] 0116 02:59:51.704865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.711548 (+ 6683us) service_pool.cc:225] Handling call
0116 02:59:52.966981 (+1255433us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b540033c7d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:52.989623 (+ 22642us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":978043136}
I20250116 02:59:52.956214 23697 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:52.928789 23700 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:53.025197 20752 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:52.918382 20720 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:53.010385 20755 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
W20250116 02:59:53.029989 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 111770us
I20250116 02:59:53.032409 20720 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:53.030723 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:53.009121 23792 raft_consensus.cc:738] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:53.035560 21332 raft_consensus.cc:1235] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 8. Current term is 9. Ops: []
I20250116 02:59:53.036199 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:53.017937 23561 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.022s sys 0.058s
I20250116 02:59:53.037927 20720 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:53.037951 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 592) took 3262 ms. Trace:
I20250116 02:59:53.042192 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:53.038551 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:53.011325 23691 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:53.042695 21332 rpcz_store.cc:276] 0116 02:59:49.779745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:49.784122 (+ 4377us) service_pool.cc:225] Handling call
0116 02:59:53.034998 (+3250876us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:53.035322 (+ 324us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:53.037635 (+ 2313us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.037886 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47668480}
W20250116 02:59:53.048960 23676 log.cc:926] Time spent T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 6.108s user 0.027s sys 0.057s
I20250116 02:59:53.048452 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:53.053112 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 276682us
I20250116 02:59:53.028365 23700 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:53.041208 23713 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:53.023609 23805 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:52.996035 23605 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:53.054932 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:53.041767 23561 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s user 0.022s sys 0.058s
I20250116 02:59:53.044952 23681 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:53.042887 20755 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:53.045142 23793 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:53.058779 23700 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:53.041118 23601 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20250116 02:59:53.058477 23713 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:53.049908 23691 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:53.067337 23792 consensus_queue.cc:260] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:53.093703 23792 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:53.085104 23713 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:53.074205 23793 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:53.105553 23793 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:53.107192 23682 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:53.108397 23605 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:53.107270 23760 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:53.111372 23760 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:53.123749 23616 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:53.114573 23689 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:53.134744 23713 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:53.137120 23713 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:53.139283 23691 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:53.169591 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.038s sys 0.089s
W20250116 02:59:53.266244 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.851s user 0.039s sys 0.090s
W20250116 02:59:53.000066 20608 consensus_meta.cc:287] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.880s user 0.011s sys 0.000s
W20250116 02:59:53.170483 20572 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.024s sys 0.026s
W20250116 02:59:53.269811 20572 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.839s user 0.025s sys 0.026s
W20250116 02:59:53.270579 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 840032us
W20250116 02:59:53.172888 20607 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.011s sys 0.040s
I20250116 02:59:53.015973 20647 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:53.274680 23559 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:53.195129 20675 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.815s user 0.042s sys 0.044s
I20250116 02:59:53.201218 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:53.276471 20607 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.861s user 0.012s sys 0.041s
I20250116 02:59:53.275825 23559 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:53.276746 20749 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:53.143932 23810 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:53.221320 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 142117us
I20250116 02:59:53.280769 23810 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:53.281183 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 274422us
I20250116 02:59:53.141875 23700 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:53.283775 23616 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:53.284384 23561 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:53.284935 23561 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:53.285372 23561 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:53.285629 23700 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:53.137053 23824 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8.
W20250116 02:59:53.289212 20675 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.909s user 0.043s sys 0.045s
I20250116 02:59:53.289669 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.256932 23601 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:53.306253 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.306994 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.309067 20601 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:53.311057 20600 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:53.328876 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:53.331086 20677 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:53.331606 20676 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:53.337389 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:53.341485 21335 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 8. Current term is 9. Ops: []
W20250116 02:59:53.342839 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 74920.2us
I20250116 02:59:53.343686 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.348999 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.342140 23559 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:53.351935 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 304533us
I20250116 02:59:53.337256 20647 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:53.357492 23792 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250116 02:59:53.366101 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:53.368430 21332 raft_consensus.cc:2388] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:53.370648 23700 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:53.371328 23700 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:53.371455 20720 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:53.371837 23700 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:53.373992 23107 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:53.378038 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:53.362769 20751 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:53.381696 20571 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:53.383388 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:53.384675 20751 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:53.384852 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:53.380125 23792 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 02:59:53.380575 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:53.394887 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 49622.8us
I20250116 02:59:53.388558 21335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 974) took 1131 ms. Trace:
I20250116 02:59:53.388221 23792 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:53.416805 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2327) took 1424 ms. Trace:
I20250116 02:59:53.426005 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:53.427012 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.429427 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:53.429731 23558 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:53.431061 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.431272 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:53.431614 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:53.432493 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.432662 20676 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:53.433339 20602 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
W20250116 02:59:53.434841 23799 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20250116 02:59:53.426971 20604 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:53.437673 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:53.422403 20752 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:53.431993 20600 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 02:59:53.434484 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:53.447999 21300 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:53.431082 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:53.453639 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:53.454515 20601 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:53.424578 23792 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Advancing to term 9
I20250116 02:59:53.456606 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2322) took 1463 ms. Trace:
I20250116 02:59:53.465744 21332 raft_consensus.cc:1235] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:53.424846 20607 rpcz_store.cc:276] 0116 02:59:51.980964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.981881 (+ 917us) service_pool.cc:225] Handling call
0116 02:59:51.982446 (+ 565us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:51.982744 (+ 298us) raft_consensus.cc:1529] Early marking committed up to index 9
0116 02:59:52.358042 (+375298us) log.cc:844] Serialized 14 byte log entry
0116 02:59:53.328238 (+970196us) log.cc:844] Serialized 14 byte log entry
0116 02:59:53.371947 (+ 43709us) log.cc:844] Serialized 14 byte log entry
0116 02:59:53.391414 (+ 19467us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:53.403593 (+ 12179us) raft_consensus.cc:1640] Marking committed up to 9
0116 02:59:53.403725 (+ 132us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.403955 (+ 230us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.405006 (+ 1051us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105875,"prepare.queue_time_us":770927,"prepare.run_cpu_time_us":158,"prepare.run_wall_time_us":164,"raft.queue_time_us":147554,"raft.run_cpu_time_us":465,"raft.run_wall_time_us":477,"thread_start_us":1221830,"threads_started":3}
I20250116 02:59:53.467670 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:53.415367 21335 rpcz_store.cc:276] 0116 02:59:52.224199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:52.225878 (+ 1679us) service_pool.cc:225] Handling call
0116 02:59:53.338827 (+1112949us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:53.339383 (+ 556us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:53.354222 (+ 14839us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.355651 (+ 1429us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37548288}
W20250116 02:59:53.467409 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 85983.7us
I20250116 02:59:53.482779 23107 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:53.483731 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:53.487242 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.487605 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.488583 20602 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 02:59:53.482810 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:53.493245 20571 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:53.493669 23742 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:53.458496 20608 rpcz_store.cc:276] 0116 02:59:51.956603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.980893 (+ 24290us) service_pool.cc:225] Handling call
0116 02:59:51.981334 (+ 441us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:53.270823 (+1289489us) raft_consensus.cc:1529] Early marking committed up to index 7
0116 02:59:53.294482 (+ 23659us) log.cc:844] Serialized 14 byte log entry
0116 02:59:53.305251 (+ 10769us) log.cc:844] Serialized 14 byte log entry
0116 02:59:53.352250 (+ 46999us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:53.352311 (+ 61us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:53.352379 (+ 68us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.357641 (+ 5262us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.420199 (+ 62558us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46709,"prepare.queue_time_us":66248,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":117,"raft.queue_time_us":51585,"raft.run_cpu_time_us":466,"raft.run_wall_time_us":2532,"thread_start_us":31717,"threads_started":2,"wal-append.queue_time_us":10489}
I20250116 02:59:53.497611 20604 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 7.
I20250116 02:59:53.497849 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 647) took 1817 ms. Trace:
I20250116 02:59:53.498456 20600 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:53.498199 23492 consensus_queue.cc:1046] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20250116 02:59:53.500926 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 53377.1us
I20250116 02:59:53.494338 23742 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:53.502864 23742 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:53.503240 23586 consensus_queue.cc:1046] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250116 02:59:53.505771 21336 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
W20250116 02:59:53.508504 23860 log.cc:926] Time spent T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20250116 02:59:53.511873 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.457597 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:53.514389 20677 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:53.501060 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 652) took 1797 ms. Trace:
I20250116 02:59:53.505687 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 650) took 1823 ms. Trace:
I20250116 02:59:53.513262 21330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 975) took 1251 ms. Trace:
I20250116 02:59:53.531136 23838 ts_tablet_manager.cc:927] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:53.521404 20753 rpcz_store.cc:276] 0116 02:59:51.703730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.703931 (+ 201us) service_pool.cc:225] Handling call
0116 02:59:53.500625 (+1796694us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b5400370ad4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:53.500813 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":289113344}
I20250116 02:59:53.514307 20750 rpcz_store.cc:276] 0116 02:59:51.680536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.682548 (+ 2012us) service_pool.cc:225] Handling call
0116 02:59:51.684831 (+ 2283us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:51.939389 (+254558us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:51.939423 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:51.939466 (+ 43us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:51.939512 (+ 46us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.497278 (+1557766us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400397454. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:53.497347 (+ 69us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.497635 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":584803712}
I20250116 02:59:53.528427 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2098) took 1336 ms. Trace:
I20250116 02:59:53.538821 20601 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:53.523324 20756 rpcz_store.cc:276] 0116 02:59:51.682043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.682772 (+ 729us) service_pool.cc:225] Handling call
0116 02:59:51.683153 (+ 381us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:51.696581 (+ 13428us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:51.696608 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:51.696647 (+ 39us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:51.696678 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.505153 (+1808475us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b54001d2254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:53.505228 (+ 75us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.505491 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30412544}
W20250116 02:59:53.580101 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 156123us
I20250116 02:59:53.513175 23561 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:53.603969 20751 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:53.621372 23581 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:53.623173 23626 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:53.623412 23858 log.cc:926] Time spent T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.318s user 0.002s sys 0.004s
W20250116 02:59:53.625044 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 91754us
I20250116 02:59:53.622359 23581 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:53.651654 23581 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:53.645682 23671 consensus_queue.cc:1046] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 02:59:53.661814 23561 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:53.668416 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2115) took 1268 ms. Trace:
I20250116 02:59:53.530427 21330 rpcz_store.cc:276] 0116 02:59:52.261693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:52.265470 (+ 3777us) service_pool.cc:225] Handling call
0116 02:59:53.511680 (+1246210us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98d0e 0000000000353db9 00007fc32bbf8a70 00007fc32bcbe834 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:53.511979 (+ 299us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:53.512567 (+ 588us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:53.512601 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:53.512686 (+ 85us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:53.512763 (+ 77us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.512846 (+ 83us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.513045 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8700,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":10424,"spinlock_wait_cycles":45623168}
W20250116 02:59:53.674571 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 65360.5us
I20250116 02:59:53.893343 20751 rpcz_store.cc:276] 0116 02:59:52.399438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.625936 (+1226498us) service_pool.cc:225] Handling call
0116 02:59:53.666639 (+ 40703us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:53.667208 (+ 569us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:53.667235 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:53.667288 (+ 53us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:53.667332 (+ 44us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.667703 (+ 371us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.667931 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:53.889573 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 396459us
W20250116 02:59:53.675647 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 141082us
I20250116 02:59:53.925767 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:53.925769 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:53.859953 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2109) took 1492 ms. Trace:
I20250116 02:59:53.857898 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2108) took 1490 ms. Trace:
I20250116 02:59:53.929327 20756 rpcz_store.cc:276] 0116 02:59:52.367584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.580095 (+1212511us) service_pool.cc:225] Handling call
0116 02:59:53.859202 (+279107us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:53.859575 (+ 373us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:53.859610 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:53.859653 (+ 43us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:53.859695 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.859750 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.859880 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:53.953099 23580 consensus_meta.cc:287] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.281s user 0.014s sys 0.010s
I20250116 02:59:53.953956 23810 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:53.671330 23792 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:53.954803 23810 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:53.956600 23792 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:53.959080 20605 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 9: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:53.676945 23823 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 02:59:53.953806 23580 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:53.936365 20750 rpcz_store.cc:276] 0116 02:59:52.367319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.579815 (+1212496us) service_pool.cc:225] Handling call
0116 02:59:53.856928 (+277113us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:53.857420 (+ 492us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:53.857448 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:53.857494 (+ 46us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:53.857532 (+ 38us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.857590 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.857772 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:53.966054 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 105793us
W20250116 02:59:53.999310 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 403283us
I20250116 02:59:53.963341 23626 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:54.003261 23626 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:54.005908 23626 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:54.008014 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56664 (request call id 978) took 1638 ms. Trace:
I20250116 02:59:54.008415 21336 rpcz_store.cc:276] 0116 02:59:52.369722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:52.395764 (+ 26042us) service_pool.cc:225] Handling call
0116 02:59:54.007762 (+1611998us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b5400770454. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:54.007930 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1942002048}
I20250116 02:59:54.011125 20571 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:54.013269 20571 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:54.011447 21331 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250116 02:59:54.017491 23792 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:54.018556 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 71385us
I20250116 02:59:54.019301 20608 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:54.024106 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:53.841799 23700 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:53.972265 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 52709.9us
I20250116 02:59:54.028628 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2091) took 1734 ms. Trace:
I20250116 02:59:54.021076 23626 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 02:59:54.046113 20750 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:54.033569 23799 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:54.033190 20755 rpcz_store.cc:276] 0116 02:59:52.161241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.196601 (+1035360us) service_pool.cc:225] Handling call
0116 02:59:53.197820 (+ 1219us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:53.260879 (+ 63059us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:53.260919 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:53.343035 (+ 82116us) log.cc:844] Serialized 26 byte log entry
0116 02:59:53.452826 (+109791us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:53.535148 (+ 82322us) log.cc:844] Serialized 14 byte log entry
0116 02:59:53.895412 (+360264us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.895514 (+ 102us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:53.895567 (+ 53us) raft_consensus.cc:1680] finished
0116 02:59:53.895578 (+ 11us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.895741 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80936,"prepare.queue_time_us":57932,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":38,"raft.queue_time_us":99190,"raft.run_cpu_time_us":581,"raft.run_wall_time_us":16088,"thread_start_us":176984,"threads_started":3,"wal-append.queue_time_us":110525}
I20250116 02:59:54.066990 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:54.032555 20572 connection.cc:232] RPC call timeout handler was delayed by -1.0283337849999725s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
W20250116 02:59:54.105885 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 49001.1us
I20250116 02:59:54.103891 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.070590 23810 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:54.110350 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.072490 23580 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:54.117945 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2092) took 1952 ms. Trace:
I20250116 02:59:54.120990 23683 raft_consensus.cc:3049] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:54.121681 23683 raft_consensus.cc:738] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:54.120276 20757 rpcz_store.cc:276] 0116 02:59:52.161514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.207205 (+1045691us) service_pool.cc:225] Handling call
0116 02:59:53.358862 (+151657us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:53.387551 (+ 28689us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:53.387581 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:53.387623 (+ 42us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:53.387665 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.387749 (+ 84us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.112328 (+724579us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54001d2750. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:54.113701 (+ 1373us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":331433344}
W20250116 02:59:54.124430 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 111352us
I20250116 02:59:54.145607 20750 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:54.132138 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:54.151541 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 368150us
I20250116 02:59:54.157323 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.126765 23799 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:54.168246 23700 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:54.166703 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:59:54.168716 23799 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:54.168768 20607 raft_consensus.cc:1270] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250116 02:59:53.539038 20749 rpcz_store.cc:276] 0116 02:59:52.182640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.277908 (+1095268us) service_pool.cc:225] Handling call
0116 02:59:53.278532 (+ 624us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:53.337771 (+ 59239us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:53.337801 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:53.338666 (+ 865us) log.cc:844] Serialized 26 byte log entry
0116 02:59:53.424976 (+ 86310us) raft_consensus.cc:1640] Marking committed up to 4
0116 02:59:53.511116 (+ 86140us) log.cc:844] Serialized 14 byte log entry
0116 02:59:53.518330 (+ 7214us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.518418 (+ 88us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:53.518460 (+ 42us) raft_consensus.cc:1680] finished
0116 02:59:53.518473 (+ 13us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:53.518646 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3065,"prepare.queue_time_us":85413,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":7014,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":13102,"thread_start_us":84865,"threads_started":1,"wal-append.queue_time_us":109205}
I20250116 02:59:54.172708 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.173962 20749 raft_consensus.cc:2388] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 7: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:54.188323 23683 consensus_queue.cc:260] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.189985 23683 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:54.193143 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2104) took 1826 ms. Trace:
I20250116 02:59:54.151549 20755 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:54.202859 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2105) took 1836 ms. Trace:
I20250116 02:59:54.203603 20753 rpcz_store.cc:276] 0116 02:59:52.366511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.547804 (+1181293us) service_pool.cc:225] Handling call
0116 02:59:54.032962 (+485158us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.202193 (+169231us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.202228 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:54.202284 (+ 56us) raft_consensus.cc:1640] Marking committed up to 2
0116 02:59:54.202326 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.202406 (+ 80us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.202746 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:54.164453 20750 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:54.210014 23696 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:54.206784 20757 raft_consensus.cc:2371] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6. Re-sending same reply.
I20250116 02:59:54.193674 20752 rpcz_store.cc:276] 0116 02:59:52.366242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.468799 (+1102557us) service_pool.cc:225] Handling call
0116 02:59:53.469270 (+ 471us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:53.469881 (+ 611us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:53.469913 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:53.470581 (+ 668us) log.cc:844] Serialized 52 byte log entry
0116 02:59:53.529630 (+ 59049us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:53.610290 (+ 80660us) log.cc:844] Serialized 14 byte log entry
0116 02:59:53.923637 (+313347us) log.cc:844] Serialized 14 byte log entry
0116 02:59:53.927614 (+ 3977us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:53.927689 (+ 75us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:54.192827 (+265138us) raft_consensus.cc:1680] finished
0116 02:59:54.192852 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.193053 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":335719,"prepare.queue_time_us":402032,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":113514,"raft.run_cpu_time_us":533,"raft.run_wall_time_us":117769,"spinlock_wait_cycles":7665024,"thread_start_us":21528,"threads_started":1,"wal-append.queue_time_us":623894}
I20250116 02:59:54.215269 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:54.231314 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.234233 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:54.232859 23799 raft_consensus.cc:2758] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20250116 02:59:54.265635 23742 consensus_meta.cc:287] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.762s user 0.005s sys 0.005s
I20250116 02:59:54.233040 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.266772 20757 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:54.267273 20753 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 2.
I20250116 02:59:54.268292 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.268952 20753 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:54.269922 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.270851 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:54.272042 20752 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:54.273769 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:54.216897 23696 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:54.291569 20755 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:54.292712 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.301039 20606 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250116 02:59:54.301983 20753 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:54.303143 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:54.305109 20723 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.043s sys 0.085s
W20250116 02:59:54.305565 20723 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.625s user 0.044s sys 0.085s
W20250116 02:59:54.305855 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 625947us
I20250116 02:59:54.308422 23822 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:54.309069 23822 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:54.309557 23862 log.cc:926] Time spent T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20250116 02:59:54.310873 23107 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:54.311576 23107 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:54.315268 20602 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 8. Current term is 9. Ops: []
I20250116 02:59:54.320683 21330 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 8. Current term is 9. Ops: []
I20250116 02:59:54.326908 23581 raft_consensus.cc:3049] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:54.327565 23581 raft_consensus.cc:738] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:54.329250 21327 raft_consensus.cc:1235] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:54.328727 23581 consensus_queue.cc:260] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.332212 23581 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
W20250116 02:59:54.334698 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 182949us
I20250116 02:59:54.170889 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:59:54.335863 20571 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:54.290958 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.294216 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:54.355247 23696 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:54.349471 20749 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:54.336694 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 654) took 2632 ms. Trace:
I20250116 02:59:54.370091 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:54.355801 23812 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:54.322507 20723 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:54.266184 23742 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.374650 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:54.377563 20755 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:54.379909 20750 raft_consensus.cc:2463] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:54.368443 23823 raft_consensus.cc:2743] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:54.387075 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 51422.1us
I20250116 02:59:54.389182 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:54.390700 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 327139us
I20250116 02:59:54.370035 20748 rpcz_store.cc:276] 0116 02:59:51.704553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:51.710815 (+ 6262us) service_pool.cc:225] Handling call
0116 02:59:51.711252 (+ 437us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:51.741901 (+ 30649us) raft_consensus.cc:1529] Early marking committed up to index 7
0116 02:59:52.125925 (+384024us) log.cc:844] Serialized 14 byte log entry
0116 02:59:52.595039 (+469114us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:52.595106 (+ 67us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:52.595161 (+ 55us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.335315 (+1740154us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b1800332740. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323da3ec6 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9
0116 02:59:54.336155 (+ 840us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b54003714d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:54.336211 (+ 56us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.336465 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1270805,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":151,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":24037,"spinlock_wait_cycles":76502016,"thread_start_us":838718,"threads_started":2,"wal-append.queue_time_us":914664,"wal-append.run_cpu_time_us":2540,"wal-append.run_wall_time_us":1049044}
I20250116 02:59:54.419965 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:54.422580 20608 raft_consensus.cc:1235] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:54.424041 20749 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:54.425117 20757 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:54.429332 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.396600 23107 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:54.424851 20750 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:54.433938 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:54.430332 20748 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:54.436617 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:54.437458 20757 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:54.399113 23565 consensus_queue.cc:1046] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
I20250116 02:59:54.444006 20753 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:54.441627 21299 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:54.446744 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:54.450625 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:54.461751 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 318058us
I20250116 02:59:54.463372 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 669) took 1674 ms. Trace:
I20250116 02:59:54.463881 20748 rpcz_store.cc:276] 0116 02:59:52.784923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.439716 (+1654793us) service_pool.cc:225] Handling call
0116 02:59:54.450107 (+ 10391us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.458651 (+ 8544us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.458686 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:54.458736 (+ 50us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:54.458765 (+ 29us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.458831 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.458986 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408}
I20250116 02:59:54.467617 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 02:59:54.468132 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 423049us
I20250116 02:59:54.471664 20750 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250116 02:59:54.471810 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 671) took 1636 ms. Trace:
I20250116 02:59:54.473465 20757 rpcz_store.cc:276] 0116 02:59:52.835706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.446847 (+1611141us) service_pool.cc:225] Handling call
0116 02:59:54.450543 (+ 3696us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.467866 (+ 17323us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.467898 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:54.471507 (+ 3609us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:54.471564 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.471644 (+ 80us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.471756 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1062144}
I20250116 02:59:54.469004 20608 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:54.472595 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1685) took 1999 ms. Trace:
I20250116 02:59:54.475236 20750 rpcz_store.cc:276] 0116 02:59:52.473196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.442314 (+1969118us) service_pool.cc:225] Handling call
0116 02:59:54.461899 (+ 19585us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.472327 (+ 10428us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.472533 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:54.477253 21329 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20250116 02:59:54.506774 21300 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:54.502504 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:54.502465 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 516121us
W20250116 02:59:54.514132 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 48350.5us
W20250116 02:59:54.522101 20572 leader_election.cc:336] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.145s (SENT)
I20250116 02:59:54.569726 20572 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:54.575183 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 142135us
W20250116 02:59:54.522454 23838 thread.cc:635] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 0.989s user 0.032s sys 0.017s
W20250116 02:59:54.578037 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 255636us
I20250116 02:59:54.515475 23742 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:54.581276 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:54.579348 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 675) took 1373 ms. Trace:
I20250116 02:59:54.599494 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 672) took 1763 ms. Trace:
I20250116 02:59:54.600090 20748 rpcz_store.cc:276] 0116 02:59:52.836036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.467955 (+1631919us) service_pool.cc:225] Handling call
0116 02:59:54.515794 (+ 47839us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.578427 (+ 62633us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.578454 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:54.599132 (+ 20678us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:54.599195 (+ 63us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.599269 (+ 74us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.599407 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:54.583730 20750 rpcz_store.cc:276] 0116 02:59:53.199216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.476147 (+1276931us) service_pool.cc:225] Handling call
0116 02:59:54.515992 (+ 39845us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.571890 (+ 55898us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.571916 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:54.571967 (+ 51us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:54.572002 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.572057 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.572222 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:54.581836 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 674) took 1627 ms. Trace:
I20250116 02:59:54.604897 20571 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:54.605703 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 74600.8us
I20250116 02:59:54.604311 20757 rpcz_store.cc:276] 0116 02:59:52.949238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.474600 (+1525362us) service_pool.cc:225] Handling call
0116 02:59:54.518832 (+ 44232us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.570661 (+ 51829us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.570697 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:54.576521 (+ 5824us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:54.576566 (+ 45us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.576631 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.576764 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:59:54.582146 23838 thread.cc:601] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 1.047s user 0.032s sys 0.017s
W20250116 02:59:54.609163 20647 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.517s (SENT)
I20250116 02:59:54.607540 23854 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:54.613240 20647 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:54.619422 20748 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:54.620222 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 676) took 1420 ms. Trace:
I20250116 02:59:54.620771 20748 rpcz_store.cc:276] 0116 02:59:53.199865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.606144 (+1406279us) service_pool.cc:225] Handling call
0116 02:59:54.614945 (+ 8801us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.619942 (+ 4997us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.620162 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:54.523890 23737 consensus_queue.cc:1035] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.089s
I20250116 02:59:54.622452 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:54.609627 23677 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:54.629755 21299 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:54.634218 23838 tablet_copy_client.cc:323] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.238.130:45223
I20250116 02:59:54.614143 23879 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250116 02:59:54.638484 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 121334us
I20250116 02:59:54.645576 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 677) took 1334 ms. Trace:
I20250116 02:59:54.649284 20750 rpcz_store.cc:276] 0116 02:59:53.311337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.602310 (+1290973us) service_pool.cc:225] Handling call
0116 02:59:54.629142 (+ 26832us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.645206 (+ 16064us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.645234 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:54.645299 (+ 65us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:54.645338 (+ 39us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.645407 (+ 69us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.645518 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:54.646186 23581 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:54.646445 20757 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:54.648767 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:54.621923 23731 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:54.660535 20752 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:54.666960 23778 raft_consensus.cc:3049] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Stepping down as leader of term 8
I20250116 02:59:54.667738 23874 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:54.668241 23736 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:54.668378 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:54.668447 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 680) took 1355 ms. Trace:
I20250116 02:59:54.670476 20605 pending_rounds.cc:77] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Aborting all ops after (but not including) 7
I20250116 02:59:54.680261 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 682) took 1159 ms. Trace:
I20250116 02:59:54.658260 20723 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:54.686583 20608 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:54.685997 20757 rpcz_store.cc:276] 0116 02:59:53.312929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.606853 (+1293924us) service_pool.cc:225] Handling call
0116 02:59:54.619002 (+ 12149us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.668098 (+ 49096us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.668354 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:54.674741 21302 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:54.686280 20748 rpcz_store.cc:276] 0116 02:59:53.521148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.621431 (+1100283us) service_pool.cc:225] Handling call
0116 02:59:54.640507 (+ 19076us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.641067 (+ 560us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.641135 (+ 68us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:54.641250 (+ 115us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:54.679984 (+ 38734us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.680056 (+ 72us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.680183 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 02:59:54.686120 20605 pending_rounds.cc:99] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f: Aborting uncommitted NO_OP operation due to leader change: 8.8
W20250116 02:59:54.671597 23870 log.cc:926] Time spent T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250116 02:59:54.690475 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:54.694882 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 84403.8us
I20250116 02:59:54.690443 20605 raft_consensus.cc:2883] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:54.698686 23817 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:54.700922 20757 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:54.711371 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56662 (request call id 607) took 1926 ms. Trace:
I20250116 02:59:54.712198 21329 rpcz_store.cc:276] 0116 02:59:52.784542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:52.797687 (+ 13145us) service_pool.cc:225] Handling call
0116 02:59:53.508269 (+710582us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc822c0 00007fc32bc6d809 00007fc32bc6d2b2 00007fc325339694
0116 02:59:53.508628 (+ 359us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:54.511940 (+1003312us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.710777 (+198837us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b540058ded4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:54.711249 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2027593728}
I20250116 02:59:54.679910 23677 raft_consensus.cc:695] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:54.698665 23879 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:54.715855 23879 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:54.716572 23879 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:54.681118 23778 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:54.715945 23677 consensus_queue.cc:237] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.698788 23765 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:54.739562 20749 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:54.742734 20753 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:54.756363 20723 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:54.770404 23810 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:54.776403 23490 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
W20250116 02:59:54.777033 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 159734us
I20250116 02:59:54.837388 23746 consensus_queue.cc:1046] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250116 02:59:54.838824 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 92503.4us
I20250116 02:59:54.700189 23731 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:54.856338 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 117486us
I20250116 02:59:54.764189 23742 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250116 02:59:54.861553 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 186972us
I20250116 02:59:54.702231 23874 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.872975 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 538) took 1438 ms. Trace:
I20250116 02:59:54.849910 20750 raft_consensus.cc:2949] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } } }
I20250116 02:59:54.875561 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1672) took 2449 ms. Trace:
I20250116 02:59:54.845868 23490 raft_consensus.cc:695] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:54.877435 23490 consensus_queue.cc:237] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.878713 23731 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:54.880786 23823 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:54.881503 23823 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.882221 20679 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:54.884346 23559 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:54.885207 23559 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.888635 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:54.889581 21327 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:54.893462 20649 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:54.899822 20572 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:54.901227 20572 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:54.902135 23812 consensus_meta.cc:287] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.520s user 0.010s sys 0.000s
I20250116 02:59:54.873715 23600 raft_consensus.cc:3049] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:54.902840 23600 raft_consensus.cc:738] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:54.903771 21331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 981) took 2082 ms. Trace:
I20250116 02:59:54.910988 20723 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:54.911610 20754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1688) took 2271 ms (client timeout 1603 ms). Trace:
W20250116 02:59:54.912087 20754 rpcz_store.cc:269] 0116 02:59:52.640134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.868194 (+2228060us) service_pool.cc:212] Skipping call since client already timed out
0116 02:59:54.911386 (+ 43192us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 02:59:54.913723 23810 raft_consensus.cc:2798] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:54.914273 23810 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:54.914674 23810 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:54.919080 23872 log.cc:926] Time spent T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.315s user 0.001s sys 0.000s
I20250116 02:59:54.904229 21331 rpcz_store.cc:276] 0116 02:59:52.821692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:52.826758 (+ 5066us) service_pool.cc:225] Handling call
0116 02:59:54.010809 (+1184051us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:54.014652 (+ 3843us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.903535 (+888883us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400770450. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:54.903699 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2721735040}
W20250116 02:59:54.896371 23683 consensus_meta.cc:287] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.706s user 0.010s sys 0.006s
I20250116 02:59:54.876024 20751 rpcz_store.cc:276] 0116 02:59:52.425866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.963587 (+1537721us) service_pool.cc:225] Handling call
0116 02:59:54.000287 (+ 36700us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:54.000807 (+ 520us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.000835 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:54.001464 (+ 629us) log.cc:844] Serialized 26 byte log entry
0116 02:59:54.395267 (+393803us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:54.522408 (+127141us) log.cc:844] Serialized 14 byte log entry
0116 02:59:54.568783 (+ 46375us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.568881 (+ 98us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:54.769537 (+200656us) spinlock_profiling.cc:243] Waited 8.57 ms on lock 0x7b18002f1d60. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd67e8 00007fc323dd9cc1 00007fc323dd9c79 00007fc323dd9c09 00007fc323dd9bd1 00007fc323dd8ecd 00007fc32ce13234 00007fc32ce13069 00007fc323db8fe6
0116 02:59:54.875176 (+105639us) raft_consensus.cc:1680] finished
0116 02:59:54.875319 (+ 143us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.875486 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25223,"prepare.queue_time_us":231156,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":105293,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"spinlock_wait_cycles":19699584,"thread_start_us":474010,"threads_started":2,"wal-append.queue_time_us":512222}
I20250116 02:59:54.896865 23399 consensus_queue.cc:1046] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20250116 02:59:54.866838 23822 consensus_meta.cc:287] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.554s user 0.016s sys 0.000s
I20250116 02:59:54.920990 20646 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:54.895169 23561 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 02:59:54.951673 23678 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:54.952028 23811 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:54.952256 23678 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:54.952466 23811 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:54.952899 23811 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:54.873548 20603 rpcz_store.cc:276] 0116 02:59:53.425852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.434881 (+ 9029us) service_pool.cc:225] Handling call
0116 02:59:53.435391 (+ 510us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:53.435960 (+ 569us) raft_consensus.cc:1529] Early marking committed up to index 7
0116 02:59:53.436437 (+ 477us) log.cc:844] Serialized 14 byte log entry
0116 02:59:54.139842 (+703405us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b180010d0e0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd67e8 00007fc323dd9cc1 00007fc323dd9c79 00007fc323dd9c09 00007fc323dd9bd1 00007fc323dd8ecd 00007fc32ce13234 00007fc32ce13069 00007fc323db8fe6
0116 02:59:54.779928 (+640086us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b180010d0e0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc32699a08c 00007fc323db8bfc 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd5fd2 00007fc32698be4b 00007fc326991913 00007fc326991e2b
0116 02:59:54.863437 (+ 83509us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:54.863489 (+ 52us) raft_consensus.cc:1640] Marking committed up to 7
0116 02:59:54.863626 (+ 137us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.863705 (+ 79us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:54.863840 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65569,"prepare.queue_time_us":4055,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":117,"raft.queue_time_us":100389,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"spinlock_wait_cycles":8959360,"thread_start_us":10920,"threads_started":1,"wal-append.queue_time_us":11525}
I20250116 02:59:54.953699 23804 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:54.939697 23778 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.956980 23778 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Advancing to term 9
I20250116 02:59:54.959064 23822 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.959968 23802 raft_consensus.cc:3049] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:54.960433 23802 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:54.961462 23822 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:54.961382 23802 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:54.935425 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 63766.1us
I20250116 02:59:54.934051 20649 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:54.943454 23874 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:54.904290 23600 consensus_queue.cc:260] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:54.945681 23823 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:54.946928 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 370034us
I20250116 02:59:54.967121 20608 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:54.972646 23758 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 02:59:54.972978 23600 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:54.974126 21302 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:54.975515 21302 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:54.975993 23802 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:54.952970 23561 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.026s
I20250116 02:59:54.958940 23736 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:54.969053 23678 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:54.993882 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:54.980741 23822 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:54.989315 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:54.999559 20647 leader_election.cc:336] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.538s (SENT)
I20250116 02:59:55.006984 20647 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.009204 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:55.009089 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:55.010234 20751 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
W20250116 02:59:55.010668 20755 consensus_meta.cc:287] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.628s user 0.010s sys 0.001s
I20250116 02:59:55.011144 20755 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:54.986183 23736 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:54.986089 23854 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 02:59:55.022836 20693 tablet_copy_service.cc:140] P 231d91433a404468a44ea0a107345aeb: Received BeginTabletCopySession request for tablet 59601d2a275b4efdbbc89dbcb608871c from peer 57fcfcd5ab01438c9a4baf8a61871aa6 ({username='slave'} at 127.0.0.1:41212)
I20250116 02:59:54.918910 23884 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:54.998944 23746 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 02:59:54.981084 23810 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.062803 23884 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.064348 23696 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:55.032114 23854 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:55.065311 23884 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:55.065496 23854 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Advancing to term 9
I20250116 02:59:55.068537 20677 raft_consensus.cc:1235] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250116 02:59:55.058384 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1673) took 2632 ms. Trace:
I20250116 02:59:55.031831 23736 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:55.087875 21336 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:55.095052 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.091885 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.056048 23878 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:55.070292 20756 rpcz_store.cc:276] 0116 02:59:52.426223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:53.965559 (+1539336us) service_pool.cc:225] Handling call
0116 02:59:54.044250 (+ 78691us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:54.087378 (+ 43128us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.087411 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:54.088005 (+ 594us) log.cc:844] Serialized 26 byte log entry
0116 02:59:54.381195 (+293190us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:54.580661 (+199466us) log.cc:844] Serialized 14 byte log entry
0116 02:59:54.675127 (+ 94466us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:54.675210 (+ 83us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:55.056625 (+381415us) raft_consensus.cc:1680] finished
0116 02:59:55.056652 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:55.058294 (+ 1642us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":218992,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":54617,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":164,"thread_start_us":541893,"threads_started":3,"wal-append.queue_time_us":425943}
I20250116 02:59:55.054286 20693 tablet_copy_service.cc:161] P 231d91433a404468a44ea0a107345aeb: Beginning new tablet copy session on tablet 59601d2a275b4efdbbc89dbcb608871c from peer 57fcfcd5ab01438c9a4baf8a61871aa6 at {username='slave'} at 127.0.0.1:41212: session id = 57fcfcd5ab01438c9a4baf8a61871aa6-59601d2a275b4efdbbc89dbcb608871c
I20250116 02:59:55.020004 23822 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:55.108770 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 206178us
I20250116 02:59:55.105487 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:55.106681 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.113193 20572 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.086781 23884 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:55.119083 20649 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:55.119263 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:55.119858 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.120795 23823 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:55.121107 23886 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.098s user 0.002s sys 0.000s
I20250116 02:59:55.119310 23884 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.125324 23884 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:55.119138 20683 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:55.125845 23884 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:55.128629 23879 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:55.132139 20571 connection.cc:232] RPC call timeout handler was delayed by -1.0095851820000235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
I20250116 02:59:55.093950 23107 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:55.134774 23812 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:55.143083 23107 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.136179 23893 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:55.135335 23889 consensus_queue.cc:1046] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20250116 02:59:55.136363 23878 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:55.146075 23889 raft_consensus.cc:3049] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:55.143422 23812 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.155642 23822 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:55.157342 23812 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.159395 23878 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:55.161092 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:35534 (request call id 9) took 1961 ms. Trace:
I20250116 02:59:55.161545 20752 rpcz_store.cc:276] 0116 02:59:53.200042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.721695 (+1521653us) service_pool.cc:225] Handling call
0116 02:59:55.160743 (+439048us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b540033ccd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d157c5 00007fc32bc6ef49 00007fc325340714 00007fc3253406a2 00007fc3253405d1 00007fc32534054c 00007fc32533f802 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:55.160970 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":614316288}
I20250116 02:59:55.138489 23810 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:55.163499 20554 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 17446c8dacf04461acb843547665250f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } }
I20250116 02:59:55.134567 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.144291 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:55.168769 23559 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.168917 23894 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250116 02:59:55.168944 21336 raft_consensus.cc:2388] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 9: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:55.172830 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:55.173662 21330 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:55.174954 20646 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:55.091612 23765 raft_consensus.cc:695] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:55.136683 20572 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:55.143447 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 153215us
I20250116 02:59:55.181526 23696 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:55.182109 23696 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:55.182457 23696 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:55.134958 23804 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:55.144963 23823 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.183622 23804 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.185025 23823 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.151762 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.185860 23804 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:55.121178 23565 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:55.153157 23690 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election won for term 8
I20250116 02:59:55.186775 23690 raft_consensus.cc:695] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:55.186676 23565 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:55.188078 23690 consensus_queue.cc:237] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.189256 20647 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.190152 23561 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:55.191500 23854 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:55.187222 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 50653.8us
I20250116 02:59:55.195458 20751 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 02:59:55.177875 23765 consensus_queue.cc:237] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.199453 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2119) took 1244 ms. Trace:
I20250116 02:59:55.202316 20750 rpcz_store.cc:276] 0116 02:59:53.955154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:54.676336 (+721182us) service_pool.cc:225] Handling call
0116 02:59:54.689452 (+ 13116us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:54.704816 (+ 15364us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:54.704841 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:54.705922 (+ 1081us) log.cc:844] Serialized 513 byte log entry
0116 02:59:54.845007 (+139085us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:55.004457 (+159450us) log.cc:844] Serialized 14 byte log entry
0116 02:59:55.199121 (+194664us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:55.199214 (+ 93us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:55.199265 (+ 51us) raft_consensus.cc:1680] finished
0116 02:59:55.199278 (+ 13us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:55.199400 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10603,"raft.queue_time_us":42857,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":26026,"thread_start_us":310721,"threads_started":2,"wal-append.queue_time_us":302936}
I20250116 02:59:55.215760 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.216125 20647 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.214267 21331 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 7.
W20250116 02:59:55.172101 23875 log.cc:926] Time spent T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.562s user 0.002s sys 0.000s
I20250116 02:59:55.220695 21300 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:55.189507 23565 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:55.181495 23879 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.146492 23889 raft_consensus.cc:738] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:55.207154 23804 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:55.126408 23884 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.243924 20572 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.247841 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:55.238430 23746 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election won for term 8
I20250116 02:59:55.239610 23854 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:55.260668 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.260974 23728 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:55.261652 20676 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:55.262210 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.262885 20682 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:55.263803 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.264219 23804 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:55.268393 23736 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.276283 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:55.277163 20693 tablet_copy_source_session.cc:215] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250116 02:59:55.277987 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:55.278790 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.280829 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:55.251394 21302 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.893s (SENT) [suppressed 36 similar messages]
W20250116 02:59:55.295423 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 79387.9us
I20250116 02:59:55.261723 23728 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.297101 23107 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:55.298271 23728 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.298779 23884 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.295645 20751 raft_consensus.cc:3049] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:55.304023 20751 raft_consensus.cc:738] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:55.300325 20757 raft_consensus.cc:1235] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:55.297451 23838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59601d2a275b4efdbbc89dbcb608871c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:59:55.304250 20677 raft_consensus.cc:1235] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:55.309880 20647 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:55.311937 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 80152.1us
I20250116 02:59:55.316097 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:55.316574 20680 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:55.319342 20647 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.321149 20647 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.321913 23762 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:55.323271 23696 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.326006 23696 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.325794 20751 consensus_queue.cc:260] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.295264 23746 raft_consensus.cc:695] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:55.346349 20751 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:55.312889 23561 raft_consensus.cc:2743] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:55.341327 23737 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:55.338495 23762 raft_consensus.cc:695] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:55.362294 20571 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:55.348191 23559 raft_consensus.cc:2798] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:55.364634 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.369961 20675 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 9.
I20250116 02:59:55.291258 23889 consensus_queue.cc:260] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.310988 23678 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.366573 23804 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:55.365969 23895 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:55.374459 23804 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:55.374917 23804 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:55.375533 23678 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.345345 23600 consensus_queue.cc:1046] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20250116 02:59:55.367120 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:55.381418 21327 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 7.
I20250116 02:59:55.359182 23737 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:55.359762 23736 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:55.365976 23559 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:55.393659 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:55.402297 21335 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:55.397174 20751 raft_consensus.cc:1270] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250116 02:59:55.367515 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:55.399792 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:55.400223 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:55.400417 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.400791 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:55.417816 20604 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:55.392554 23737 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:55.437044 23874 raft_consensus.cc:3049] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:55.437628 23874 raft_consensus.cc:738] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:55.383709 23760 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250116 02:59:55.409222 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 88165.4us
I20250116 02:59:55.406630 20751 pending_rounds.cc:77] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 6
I20250116 02:59:55.373594 23812 consensus_queue.cc:1046] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20250116 02:59:55.432106 23893 consensus_queue.cc:1046] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 02:59:55.436415 21330 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:55.456408 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 84034.5us
I20250116 02:59:55.448864 20751 pending_rounds.cc:99] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250116 02:59:55.457510 20751 raft_consensus.cc:2883] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250116 02:59:55.460980 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 111734us
I20250116 02:59:55.430701 23873 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
I20250116 02:59:55.464043 20647 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.403793 23559 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:55.471314 21329 raft_consensus.cc:1235] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 8. Current term is 9. Ops: []
I20250116 02:59:55.372483 23823 consensus_queue.cc:1046] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20250116 02:59:55.493031 23823 consensus_queue.cc:1225] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer 57fcfcd5ab01438c9a4baf8a61871aa6 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250116 02:59:55.493515 23893 raft_consensus.cc:3049] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:55.493903 23893 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:55.495857 23600 consensus_queue.cc:1035] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250116 02:59:55.498013 23736 consensus_queue.cc:1046] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250116 02:59:55.499260 23736 raft_consensus.cc:3049] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Stepping down as leader of term 8
I20250116 02:59:55.499610 23736 raft_consensus.cc:738] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:55.500560 23736 consensus_queue.cc:260] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.502068 23736 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Advancing to term 9
I20250116 02:59:55.473241 20649 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:55.477931 20754 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250116 02:59:55.505332 20571 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.505824 23903 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 02:59:55.506762 23823 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader election won for term 9
I20250116 02:59:55.507378 23823 raft_consensus.cc:695] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 9 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:55.508167 20752 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250116 02:59:55.494853 23893 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.510953 20678 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:55.375061 23746 consensus_queue.cc:237] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.386116 23895 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.450109 23884 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:55.480542 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:55.482861 23894 raft_consensus.cc:3049] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Stepping down as leader of term 7
I20250116 02:59:55.484138 20756 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20250116 02:59:55.485230 20554 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 27fcf38273074747a64f8e0c6d42de37, TS 231d91433a404468a44ea0a107345aeb -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:55.486250 20683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1765) took 3113 ms (client timeout 2177 ms). Trace:
W20250116 02:59:55.486562 23899 log.cc:926] Time spent T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250116 02:59:55.487546 23731 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:55.487861 20647 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.490954 23762 consensus_queue.cc:237] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.492460 20755 raft_consensus.cc:1235] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20250116 02:59:55.479156 20503 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41972 (request call id 336) took 18170 ms (client timeout 15000 ms). Trace:
I20250116 02:59:55.511350 20572 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:55.507311 20748 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:55.513584 20757 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:55.529448 23731 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:55.535789 23731 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:55.536651 23895 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:55.515070 20720 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.449854 23760 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:55.462644 23890 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 02:59:55.528831 21335 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:55.526168 20682 raft_consensus.cc:1235] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
W20250116 02:59:55.523696 20683 rpcz_store.cc:269] 0116 02:59:52.373234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:52.375600 (+ 2366us) service_pool.cc:225] Handling call
0116 02:59:55.473352 (+3097752us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b54002fdd54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:55.486155 (+ 12803us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5617397120}
I20250116 02:59:55.522711 20750 raft_consensus.cc:2463] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
W20250116 02:59:55.523440 20554 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 27fcf38273074747a64f8e0c6d42de37, TS 231d91433a404468a44ea0a107345aeb -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.535028 23540 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.014s
I20250116 02:59:55.494853 23737 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.513798 20604 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:55.523842 20675 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:55.547140 20720 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.553416 20646 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:55.526449 23894 raft_consensus.cc:738] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:55.443114 23874 consensus_queue.cc:260] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.544064 23760 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:55.592550 20647 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:55.594300 20680 raft_consensus.cc:1235] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
W20250116 02:59:55.597323 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 50308.6us
W20250116 02:59:55.597797 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 86573.5us
I20250116 02:59:55.553216 23895 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:55.599452 20600 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 5. Ops: []
W20250116 02:59:55.592033 23811 consensus_meta.cc:287] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.639s user 0.010s sys 0.003s
I20250116 02:59:55.559657 23778 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:55.603824 20572 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.606633 23737 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.508420 23823 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.615202 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:55.546465 23873 consensus_queue.cc:1046] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20250116 02:59:55.592157 23107 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:55.592396 23678 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:55.594930 23893 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:55.546149 23884 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.632997 23894 consensus_queue.cc:260] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:55.641033 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 88389.3us
I20250116 02:59:55.638629 23758 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250116 02:59:55.654573 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 101121us
I20250116 02:59:55.676015 20649 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:55.637514 20505 catalog_manager.cc:5526] T 0b57de7b372d4101b7f88debce8eea57 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 3 to 5, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 5 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250116 02:59:55.655055 20720 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:55.646934 23879 consensus_queue.cc:1046] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20250116 02:59:55.623131 23811 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.675225 23758 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:55.684166 23758 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:55.684924 23811 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:55.687857 20647 leader_election.cc:336] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.948s (SENT)
I20250116 02:59:55.688445 20647 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:55.689440 23903 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:55.658866 23884 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:55.660584 23905 consensus_queue.cc:1046] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20250116 02:59:55.600673 23817 consensus_queue.cc:1046] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
W20250116 02:59:55.532140 20503 rpcz_store.cc:269] 0116 02:59:37.308728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:37.373998 (+ 65270us) service_pool.cc:225] Handling call
0116 02:59:38.429395 (+1055397us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54000af5d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c94d50 00007fc326d17f2a 00007fc32ca33dec 00007fc32cb2bdee 00007fc32b07efc4 00007fc32b07ef52 00007fc32b07ee81 00007fc32b07edfc 00007fc32b07e0b2 00007fc32513adfc 00007fc32513a236
0116 02:59:55.465850 (+17036455us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:46.228803 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:46.229651 (+ 848us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:46.229717 (+ 66us) write_op.cc:435] Acquired schema lock
0116 02:59:46.229738 (+ 21us) tablet.cc:659] Decoding operations
0116 02:59:46.230984 (+ 1246us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:46.231072 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:46.231106 (+ 34us) tablet.cc:682] Acquiring locks for 10 operations
0116 02:59:46.232144 (+ 1038us) tablet.cc:698] Row locks acquired
0116 02:59:46.232159 (+ 15us) write_op.cc:260] PREPARE: finished
0116 02:59:46.243214 (+ 11055us) write_op.cc:270] Start()
0116 02:59:46.243361 (+ 147us) write_op.cc:276] Timestamp: P: 1736996386243127 usec, L: 0
0116 02:59:46.243385 (+ 24us) op_driver.cc:348] REPLICATION: starting
0116 02:59:46.244071 (+ 686us) log.cc:844] Serialized 4049 byte log entry
0116 02:59:47.306835 (+1062764us) op_driver.cc:464] REPLICATION: finished
0116 02:59:48.204472 (+897637us) write_op.cc:301] APPLY: starting
0116 02:59:48.250910 (+ 46438us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:48.418012 (+167102us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:48.456439 (+ 38427us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:48.974322 (+517883us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:49.125939 (+151617us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0116 02:59:49.146085 (+ 20146us) write_op.cc:312] APPLY: finished
0116 02:59:49.320102 (+174017us) log.cc:844] Serialized 77 byte log entry
0116 02:59:49.468086 (+147984us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:49.483997 (+ 15911us) write_op.cc:454] Released schema lock
0116 02:59:49.531908 (+ 47911us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:49.532408 (+ 500us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:49.532443 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2473454,"reported_tablets":36,"spinlock_wait_cycles":785311104,"child_traces":[["op",{"apply.queue_time_us":839069,"apply.run_cpu_time_us":63589,"apply.run_wall_time_us":1340377,"num_ops":10,"prepare.queue_time_us":258372,"prepare.run_cpu_time_us":39613,"prepare.run_wall_time_us":203259,"raft.queue_time_us":319120,"raft.run_cpu_time_us":38579,"raft.run_wall_time_us":361636,"replication_time_us":1021701,"thread_start_us":813475,"threads_started":5,"wal-append.queue_time_us":456994,"wal-append.run_cpu_time_us":34770,"wal-append.run_wall_time_us":2706491}]]}
I20250116 02:59:55.610574 23765 raft_consensus.cc:2798] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:55.663635 23760 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.680020 23737 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:55.677474 23812 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:55.679867 20505 catalog_manager.cc:5526] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 4 to 5, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 5 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:55.671334 23677 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:55.679893 20675 raft_consensus.cc:1235] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:55.671404 23894 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:55.682087 23559 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.713917 23677 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.666363 23778 raft_consensus.cc:695] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:55.715543 23778 consensus_queue.cc:237] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.716136 23677 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:55.703491 23884 raft_consensus.cc:2758] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20250116 02:59:55.718422 23760 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.719908 23874 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:55.720367 20679 raft_consensus.cc:1235] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 2. Current term is 3. Ops: []
I20250116 02:59:55.693794 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:55.721971 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:55.714547 23559 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.723351 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:55.690829 23688 consensus_queue.cc:1046] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250116 02:59:55.723500 23532 raft_consensus.cc:3049] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Stepping down as leader of term 5
W20250116 02:59:55.725710 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 79520.2us
I20250116 02:59:55.693862 23889 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:55.732264 23802 raft_consensus.cc:3049] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:55.732795 23802 raft_consensus.cc:738] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:55.734930 23914 raft_consensus.cc:3049] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:55.735477 23914 raft_consensus.cc:738] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:55.736510 23914 consensus_queue.cc:260] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.738013 23914 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:55.725227 23737 raft_consensus.cc:695] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:55.739907 23737 consensus_queue.cc:237] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.697082 23765 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:55.744163 23765 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:55.745684 21330 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 02:59:55.729600 23532 raft_consensus.cc:738] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:55.747087 20646 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:55.747673 23817 raft_consensus.cc:3049] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:55.748134 23817 raft_consensus.cc:738] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:55.695250 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 91557.5us
I20250116 02:59:55.727068 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:55.755136 20750 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
W20250116 02:59:55.727231 20647 leader_election.cc:336] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.100s (SENT)
I20250116 02:59:55.756609 20647 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:55.755254 20505 catalog_manager.cc:5526] T e1b6569e42b74436b3ec62c56499eb17 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 4 to 5, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 5 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } }
I20250116 02:59:55.767539 23532 consensus_queue.cc:260] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.769438 23532 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:55.702651 23811 consensus_queue.cc:1046] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20250116 02:59:55.773784 23731 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.749087 23817 consensus_queue.cc:260] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.780155 23817 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:55.781823 21300 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 02:59:55.783931 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 83658.4us
I20250116 02:59:55.798662 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:55.801297 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 74181.7us
I20250116 02:59:55.733989 23802 consensus_queue.cc:260] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:55.823472 20554 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 81fd75864c6947f09d4e5d31a757c4fa, TS 17446c8dacf04461acb843547665250f -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:55.824003 20554 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 81fd75864c6947f09d4e5d31a757c4fa, TS 17446c8dacf04461acb843547665250f -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:55.762416 23690 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election won for term 6
W20250116 02:59:55.865989 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 66171.5us
I20250116 02:59:55.791702 23731 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.813158 23683 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:55.871510 20675 raft_consensus.cc:1235] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:55.813416 23760 raft_consensus.cc:2743] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:55.872716 20505 catalog_manager.cc:5526] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 4 to 3, leader changed from <none> to 231d91433a404468a44ea0a107345aeb (127.18.238.130). New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 02:59:55.889103 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 180485us
I20250116 02:59:55.895196 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.896122 20683 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:55.901173 23399 consensus_queue.cc:1046] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 02:59:55.904345 20720 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:55.876257 23683 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:55.816581 23677 consensus_queue.cc:1046] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20250116 02:59:55.908731 20505 catalog_manager.cc:5526] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 2 to 3, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 3 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
W20250116 02:59:55.914079 20647 leader_election.cc:336] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.149s (SENT)
I20250116 02:59:55.914599 20647 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:55.916291 20720 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:55.931326 23677 raft_consensus.cc:3049] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Stepping down as leader of term 2
I20250116 02:59:55.931970 23677 raft_consensus.cc:738] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 02:59:55.931222 23917 log.cc:926] Time spent T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250116 02:59:55.898536 23690 raft_consensus.cc:695] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:55.895328 23812 raft_consensus.cc:3049] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:55.901217 23776 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:55.968027 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:55.912209 23683 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:55.913581 23804 consensus_meta.cc:287] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.536s user 0.011s sys 0.002s
I20250116 02:59:55.945504 23690 consensus_queue.cc:237] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.938228 23677 consensus_queue.cc:260] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.981204 23677 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:55.947727 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.947995 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.948228 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.948523 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.949255 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.949551 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:55.949983 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.950786 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:55.984614 20604 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:55.984972 20602 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:55.956192 20505 catalog_manager.cc:5526] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 4 to 5, leader changed from 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 5 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:55.959067 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.967837 23776 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.944687 23758 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:55.969745 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:55.932889 23896 log.cc:926] Time spent T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250116 02:59:55.868770 23915 consensus_queue.cc:1046] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.135s
I20250116 02:59:56.001596 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:56.002375 21330 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:55.978497 23804 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.003351 20505 catalog_manager.cc:5526] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } }
I20250116 02:59:56.004690 23804 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:55.961928 23812 raft_consensus.cc:738] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:56.005160 23758 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:56.005784 20680 raft_consensus.cc:1235] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:56.005442 20505 catalog_manager.cc:5526] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 3 to 6, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 17446c8dacf04461acb843547665250f (127.18.238.131). New cstate: current_term: 6 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:55.937161 23895 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:55.989809 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.983271 20603 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:55.983397 20608 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
W20250116 02:59:55.993955 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 64039.1us
I20250116 02:59:55.997001 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:56.031246 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:55.997915 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:55.998194 23776 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:55.998879 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:56.000283 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:55.998881 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:55.986493 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:56.008087 20646 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:56.031246 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:55.997777 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:56.036989 20605 raft_consensus.cc:2388] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:56.037497 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:55.971799 23927 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:56.045423 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:56.052496 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:56.054121 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 46141.5us
I20250116 02:59:56.054649 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:56.056058 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:56.056887 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:56.057050 20748 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 02:59:56.056877 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:56.002002 23915 raft_consensus.cc:3049] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Stepping down as leader of term 4
W20250116 02:59:56.058455 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 267071us
I20250116 02:59:56.064363 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:56.059664 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 145645us
I20250116 02:59:56.062376 20505 catalog_manager.cc:5526] T c881ef7597c74adfbc9e875bfb6e5f7f P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 3 to 4, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 4 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:56.065513 20602 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:56.006731 23559 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:56.067658 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:56.070255 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:56.070957 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:56.071007 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:56.067188 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:56.079185 20607 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:56.073762 20604 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:56.074187 21329 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:56.075531 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:56.081163 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:56.081447 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:56.085554 20649 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.041160 23107 raft_consensus.cc:3049] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:56.054214 23903 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:56.082113 20599 raft_consensus.cc:2371] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 17446c8dacf04461acb843547665250f in term 7. Re-sending same reply.
I20250116 02:59:56.043088 23895 raft_consensus.cc:695] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 7 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:56.097643 20603 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:56.096835 20505 catalog_manager.cc:5526] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:56.100795 21332 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:56.064167 23915 raft_consensus.cc:738] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:56.097496 20601 raft_consensus.cc:2405] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250116 02:59:56.094727 23678 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250116 02:59:56.091231 23927 raft_consensus.cc:695] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 7 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:56.097031 23903 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:56.095211 23107 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:56.126394 20676 raft_consensus.cc:1235] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:56.126529 23802 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:56.144306 23754 raft_consensus.cc:3049] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:56.163620 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:56.118136 23763 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
W20250116 02:59:56.165019 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 74404.6us
W20250116 02:59:56.165967 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 57257.3us
I20250116 02:59:56.172971 20720 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:56.179996 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:56.191253 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 67753us
I20250116 02:59:56.193271 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.194196 20755 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:56.149973 23903 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:56.164285 23754 raft_consensus.cc:738] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:56.201882 21332 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20250116 02:59:56.199973 20647 leader_election.cc:336] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.243s (SENT)
I20250116 02:59:56.207136 20647 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.208714 20676 raft_consensus.cc:1270] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20250116 02:59:56.188017 23884 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:56.215833 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:56.130162 23683 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.099833 23812 consensus_queue.cc:260] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.215920 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:56.222651 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:56.222487 20607 raft_consensus.cc:1235] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:56.223371 20602 raft_consensus.cc:2371] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7. Re-sending same reply.
I20250116 02:59:56.230206 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:56.231551 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:56.232002 20755 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:56.235235 20604 raft_consensus.cc:1235] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:56.236120 20505 catalog_manager.cc:5526] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 3 to 4, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 4 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } }
I20250116 02:59:56.271883 23927 consensus_queue.cc:237] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.348002 23589 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250116 02:59:56.349267 23874 consensus_meta.cc:287] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.629s user 0.011s sys 0.000s
I20250116 02:59:56.230327 23742 consensus_queue.cc:1046] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.121s
I20250116 02:59:56.267436 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:56.354915 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 116656us
I20250116 02:59:56.362833 20723 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:56.356406 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 136368us
I20250116 02:59:56.218408 23895 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.223063 23107 consensus_queue.cc:260] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:56.355010 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 169374us
W20250116 02:59:56.378489 23928 log.cc:926] Time spent T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.334s user 0.000s sys 0.001s
I20250116 02:59:56.194962 23915 consensus_queue.cc:260] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.383273 21302 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.211311 23894 consensus_queue.cc:1046] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20250116 02:59:56.386374 20723 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.214859 23918 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.168s
I20250116 02:59:56.389859 20603 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:56.355623 23930 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:56.402472 23914 consensus_meta.cc:287] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.664s user 0.014s sys 0.000s
I20250116 02:59:56.427127 20755 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:56.382130 23561 consensus_queue.cc:1046] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20250116 02:59:56.412478 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:56.439895 20750 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:56.423686 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 52688.5us
I20250116 02:59:56.390756 20505 catalog_manager.cc:5526] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 3 to 4, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129). New cstate: current_term: 4 leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: UNKNOWN } } }
I20250116 02:59:56.437492 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250116 02:59:56.390156 20572 leader_election.cc:336] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.603s (SENT)
I20250116 02:59:56.443068 20572 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:56.450204 21335 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
W20250116 02:59:56.463625 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 75623.5us
I20250116 02:59:56.402319 23683 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:56.475004 23909 consensus_queue.cc:1035] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20250116 02:59:56.386380 23754 consensus_queue.cc:260] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.480441 20679 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:56.483075 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:56.355010 23925 consensus_queue.cc:1046] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.084s
I20250116 02:59:56.483989 20680 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:56.484683 20505 catalog_manager.cc:5526] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "231d91433a404468a44ea0a107345aeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 02:59:56.383718 23559 consensus_queue.cc:1046] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20250116 02:59:56.437829 23532 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250116 02:59:56.494869 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 129898us
I20250116 02:59:56.429558 23940 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:56.502154 20750 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:56.504460 21300 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:56.492347 23736 raft_consensus.cc:3049] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:56.515141 20749 raft_consensus.cc:1235] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:56.515087 20505 catalog_manager.cc:5526] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f reported cstate change: term changed from 1 to 4, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 4 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 02:59:56.518729 23922 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20250116 02:59:56.511823 23736 raft_consensus.cc:738] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:56.522061 23736 consensus_queue.cc:260] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.523394 23736 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:56.526402 23561 raft_consensus.cc:3049] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:56.526923 23561 raft_consensus.cc:738] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:56.527761 23561 consensus_queue.cc:260] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.528861 23561 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:56.494366 23811 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:56.540261 20675 raft_consensus.cc:1235] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
W20250116 02:59:56.533897 23889 consensus_meta.cc:287] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.774s user 0.012s sys 0.000s
W20250116 02:59:56.548208 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 158089us
I20250116 02:59:56.548298 21302 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:56.550805 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 178516us
I20250116 02:59:56.552259 20723 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.565972 20647 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:56.566233 20723 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.568200 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:56.573938 21302 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:56.564402 23939 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.209s user 0.001s sys 0.000s
I20250116 02:59:56.580449 23873 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:56.480113 23930 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.584457 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:56.596659 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 88196.2us
W20250116 02:59:56.580900 23838 consensus_meta.cc:287] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.060s user 0.013s sys 0.001s
W20250116 02:59:56.631982 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 48109.8us
I20250116 02:59:56.555570 23540 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
W20250116 02:59:56.636022 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 69893.5us
I20250116 02:59:56.657922 20723 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.658387 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:56.632587 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 58727.8us
W20250116 02:59:56.646965 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 142593us
I20250116 02:59:56.591172 23854 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250116 02:59:56.661203 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 95338.8us
I20250116 02:59:56.661598 21302 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.663450 20647 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.567936 23940 raft_consensus.cc:2743] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:56.555882 23914 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20250116 02:59:56.687417 20604 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:56.688536 21302 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:56.694955 20603 raft_consensus.cc:1235] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250116 02:59:56.664835 20723 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:56.698808 20602 raft_consensus.cc:3049] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:56.699311 20602 raft_consensus.cc:738] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:56.590596 23760 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:56.591693 23763 raft_consensus.cc:3049] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:56.641202 23884 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:56.714768 20600 consensus_meta.cc:287] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.527s user 0.010s sys 0.000s
I20250116 02:59:56.713975 20603 raft_consensus.cc:1270] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250116 02:59:56.702831 20602 consensus_queue.cc:260] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.723946 20600 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:56.723121 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:56.724628 20602 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:56.724958 20757 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:56.694689 23927 raft_consensus.cc:2743] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:56.710217 23765 consensus_meta.cc:287] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.965s user 0.018s sys 0.002s
I20250116 02:59:56.720985 23763 raft_consensus.cc:738] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:56.734809 23949 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:56.803423 23949 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:56.804040 23949 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.737003 23930 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:56.737679 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 74311us
I20250116 02:59:56.746655 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.747539 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.816177 20749 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:56.817024 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.750417 20676 raft_consensus.cc:1235] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:56.817186 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:56.821146 20749 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:56.751602 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.752009 23838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56664 (request call id 972) took 4601 ms. Trace:
I20250116 02:59:56.708369 23540 raft_consensus.cc:695] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:56.822311 20750 raft_consensus.cc:2388] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:56.821962 23838 rpcz_store.cc:276] 0116 02:59:52.150122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:52.151165 (+ 1043us) service_pool.cc:225] Handling call
0116 02:59:53.531746 (+1380581us) ts_tablet_manager.cc:928] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
0116 02:59:54.317109 (+785363us) spinlock_profiling.cc:243] Waited 8.58 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc323db881e 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd27da 00007fc323dd2702 00007fc32bc1d671
0116 02:59:55.295889 (+978780us) tablet_copy_client.cc:368] Tablet copy session begun
0116 02:59:55.490010 (+194121us) tablet_metadata.cc:641] Metadata flushed
0116 02:59:55.490336 (+ 326us) tablet_copy_client.cc:497] Wrote new tablet metadata
0116 02:59:56.699663 (+1209327us) tablet_copy_client.cc:418] Wrote new consensus metadata
0116 02:59:56.750978 (+ 51315us) tablet_replica.cc:206] Creating consensus instance
0116 02:59:56.751579 (+ 601us) ts_tablet_manager.cc:983] Replica registered.
0116 02:59:56.751839 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":17178,"spinlock_wait_cycles":19744512,"tablet-copy.queue_time_us":662257,"thread_start_us":1566621,"threads_started":2}
I20250116 02:59:56.823122 23540 consensus_queue.cc:237] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.824091 23107 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:56.830739 23823 raft_consensus.cc:3049] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:56.831279 23823 raft_consensus.cc:738] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:56.832211 23823 consensus_queue.cc:260] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.832846 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.833611 23823 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:56.834169 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:56.753545 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.754324 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.755323 20753 raft_consensus.cc:1235] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:56.834666 23765 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.839828 20608 raft_consensus.cc:1235] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:56.702052 23758 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:56.842551 23758 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:56.843030 23758 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:56.843463 23765 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:56.844367 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:56.760089 20554 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6af80665fed7453e80094933a2b5bc2d, TS 231d91433a404468a44ea0a107345aeb -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:56.846798 20554 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6af80665fed7453e80094933a2b5bc2d, TS 231d91433a404468a44ea0a107345aeb -> TS 57fcfcd5ab01438c9a4baf8a61871aa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.855443 20757 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:56.856473 20748 raft_consensus.cc:1235] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20250116 02:59:56.762249 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.869822 20755 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:56.698194 23802 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250116 02:59:56.764542 23677 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:56.871291 23677 raft_consensus.cc:695] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:56.872210 23677 consensus_queue.cc:237] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.874924 23802 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:56.764703 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 76297.4us
I20250116 02:59:56.838039 20752 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 02:59:56.728955 20683 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:56.735193 23776 consensus_queue.cc:1035] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20250116 02:59:56.835656 20750 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:56.716393 23760 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:56.784263 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 124972us
I20250116 02:59:56.754503 20680 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:56.881289 23760 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:56.882263 20723 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.883464 20723 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.884542 20723 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.790114 20604 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250116 02:59:56.885841 23879 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:56.886519 23879 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:56.887106 23879 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:56.888491 20723 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.791613 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:56.792914 20602 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250116 02:59:56.808315 20647 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:56.893590 20602 pending_rounds.cc:77] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Aborting all ops after (but not including) 6
I20250116 02:59:56.894114 20602 pending_rounds.cc:99] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250116 02:59:56.894193 20723 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.894531 20602 raft_consensus.cc:2883] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:56.896530 20751 raft_consensus.cc:2388] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:56.723490 23940 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:56.808209 23763 consensus_queue.cc:260] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.806226 23949 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:56.913025 23949 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:56.913376 20723 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:56.913682 23927 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:59:56.913586 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 105348us
I20250116 02:59:56.915028 23763 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:56.812419 23817 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:56.832787 20749 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:56.824230 23838 tablet_copy_client.cc:806] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 0 data blocks...
W20250116 02:59:56.760917 23903 consensus_meta.cc:287] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.547s user 0.010s sys 0.005s
I20250116 02:59:56.859329 23762 consensus_queue.cc:1046] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250116 02:59:56.778144 23812 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:56.877051 20683 raft_consensus.cc:604] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.925045 21329 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250116 02:59:56.925392 20683 raft_consensus.cc:612] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:56.876538 21302 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.836565 23915 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:56.875532 23802 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.879827 23944 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:56.895123 23765 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:56.908846 23940 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:56.644209 23874 consensus_queue.cc:1046] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20250116 02:59:56.914963 20606 raft_consensus.cc:1270] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250116 02:59:56.914414 20723 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:56.944149 20677 raft_consensus.cc:1235] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:56.919625 23936 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:56.929090 20647 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
W20250116 02:59:56.914628 23937 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250116 02:59:56.923846 23879 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:56.907135 23776 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20250116 02:59:56.914417 23927 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:56.919654 23905 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 02:59:56.945824 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 69372.6us
I20250116 02:59:56.960116 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:56.961989 21302 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.945127 23925 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:56.923249 23884 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:56.951965 21329 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250116 02:59:56.914486 23731 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:56.971225 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:56.946336 23940 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:56.956218 20647 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:56.938465 20675 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:56.930629 23895 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:56.941865 23742 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:56.942997 23762 raft_consensus.cc:3049] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 02:59:56.935003 23736 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20250116 02:59:56.926208 23949 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.928443 23817 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.957140 23950 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:56.957124 23838 tablet_copy_client.cc:670] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Starting download of 1 WAL segments...
I20250116 02:59:56.931566 23903 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.979643 21302 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:57.003623 20647 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:56.945282 23765 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.971850 23731 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:56.977990 23688 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:56.980662 23561 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:57.013161 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:56.984385 23762 raft_consensus.cc:738] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:56.962982 23879 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.029803 23690 raft_consensus.cc:2743] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:57.044770 23932 raft_consensus.cc:2758] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20250116 02:59:57.046170 23948 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.046828 23954 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:57.047245 23957 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:57.039543 23731 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:56.942380 23944 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.967028 23905 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.984155 23895 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:56.995913 23742 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:57.094285 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 114733us
I20250116 02:59:56.957592 23936 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:56.971333 23925 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.107673 21329 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:36880
I20250116 02:59:57.108706 21329 raft_consensus.cc:604] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:59:57.109136 21329 raft_consensus.cc:612] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250116 02:59:56.918061 23746 consensus_queue.cc:1159] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "231d91433a404468a44ea0a107345aeb" responder_term: 7 status { last_received { term: 6 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250116 02:59:57.078374 23957 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:57.080791 23927 raft_consensus.cc:695] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:57.123752 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.127509 20602 raft_consensus.cc:1270] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20250116 02:59:57.128134 20602 pending_rounds.cc:77] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Aborting all ops after (but not including) 7
I20250116 02:59:57.128484 20602 pending_rounds.cc:99] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20250116 02:59:57.128795 20602 raft_consensus.cc:2883] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 02:59:57.100482 23754 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:57.078696 23561 consensus_queue.cc:1046] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
W20250116 02:59:57.134591 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 87539.6us
I20250116 02:59:57.140408 23879 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:57.141669 23879 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:57.143648 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.144147 20647 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:57.116356 23683 consensus_queue.cc:1159] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "17446c8dacf04461acb843547665250f" responder_term: 6 status { last_received { term: 6 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 6 index: 7 } }
I20250116 02:59:57.121941 23802 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.080389 23948 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.120700 23957 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:57.145437 20679 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:57.147336 20647 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.148080 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.182526 20599 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:57.183599 20571 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 02:59:57.159816 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 232707us
I20250116 02:59:57.183893 20723 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:57.184976 20720 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.142207 23879 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.186069 20647 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:57.163326 23955 log.cc:926] Time spent T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.214s user 0.000s sys 0.002s
I20250116 02:59:57.187525 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.115361 23954 consensus_queue.cc:1035] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20250116 02:59:57.195263 20572 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.198328 21329 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250116 02:59:57.197954 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.134752 23758 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.205513 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.207298 20681 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:57.207929 23893 raft_consensus.cc:2743] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 02:59:57.208277 23959 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
I20250116 02:59:57.208810 23914 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.209364 23914 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.211627 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:57.212568 20600 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:57.218212 20681 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250116 02:59:57.187012 23553 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250116 02:59:57.227787 20600 raft_consensus.cc:2463] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
W20250116 02:59:57.215476 23960 log.cc:926] Time spent T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250116 02:59:57.191785 23964 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:57.190129 23930 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:57.199321 23817 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.199401 23565 raft_consensus.cc:2743] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:57.268379 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 02:59:57.247414 23963 log.cc:926] Time spent T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250116 02:59:57.247871 23965 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:57.248030 23742 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.236735 23765 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.197762 23678 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250116 02:59:57.250490 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.257467 23930 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:57.190307 23950 raft_consensus.cc:695] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:57.258061 20676 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:57.258906 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.197518 23903 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.262790 23677 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.246831 23948 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:57.207686 23946 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.229900 23949 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:57.280252 20677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:57.317644 20677 raft_consensus.cc:604] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:59:57.249054 23905 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.285909 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.318822 20675 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:57.286924 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.283231 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:57.149749 23762 consensus_queue.cc:260] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.249861 23944 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:57.253757 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 58598.9us
I20250116 02:59:57.327049 20572 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:57.291193 20600 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:57.260090 23964 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:57.306864 23965 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.328505 20572 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.304246 20679 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:57.301887 23895 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:57.303402 23930 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:57.251248 23760 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.247399 23957 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:57.311847 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.290869 23914 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.347949 20571 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:57.349035 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:57.350492 21332 raft_consensus.cc:2388] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 9: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:57.330082 20572 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.324505 21299 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:57.345005 20682 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 9.
I20250116 02:59:57.236330 23927 consensus_queue.cc:237] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.354127 20571 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:57.335207 23918 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:57.354971 20680 raft_consensus.cc:1235] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:57.328454 23936 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.339699 23817 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:57.317811 23677 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.335009 23964 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:57.348754 23765 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.379165 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:57.350132 23925 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.385046 23962 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:57.325212 23946 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.368400 23874 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:57.395134 23731 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.408180 23765 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.409698 23946 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.410305 23731 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.364179 23918 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:57.368342 23950 consensus_queue.cc:237] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.383230 23948 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:57.399814 23758 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.401089 23677 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:57.401803 23974 log.cc:926] Time spent T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20250116 02:59:57.421706 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.345579 23965 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.404742 20682 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:57.407496 23962 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.403337 23879 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:57.425959 20677 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:57.428097 20682 raft_consensus.cc:604] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.356551 23956 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:57.411624 23874 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:57.434867 23874 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:57.437777 23962 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.410552 23765 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.381348 23936 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 02:59:57.439270 23936 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:57.439713 23936 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Advancing to term 9
I20250116 02:59:57.410583 21333 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:57.433562 21299 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:57.378731 23969 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.381022 23905 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:57.411464 23918 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:57.444074 23945 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20250116 02:59:57.456226 20682 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:57.450788 23905 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 02:59:57.401096 23957 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:57.456990 20682 raft_consensus.cc:604] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 02:59:57.458045 20754 raft_consensus.cc:1235] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:57.459331 20682 raft_consensus.cc:612] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 02:59:57.452565 23874 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.458900 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 02:59:57.453620 23812 consensus_meta.cc:287] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.524s user 0.010s sys 0.000s
I20250116 02:59:57.429977 23760 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.459419 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.443830 23956 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:57.453811 23677 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:57.462543 20757 raft_consensus.cc:1270] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250116 02:59:57.464272 20752 raft_consensus.cc:2371] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5. Re-sending same reply.
I20250116 02:59:57.488756 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.464763 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.489738 20755 raft_consensus.cc:2371] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6. Re-sending same reply.
I20250116 02:59:57.427675 23948 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:57.471478 23823 consensus_meta.cc:287] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.637s user 0.010s sys 0.006s
I20250116 02:59:57.490640 23893 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.486987 23677 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:57.504065 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.509690 20757 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:57.471781 23956 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:57.511921 20649 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:57.513129 20649 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:57.515254 20755 raft_consensus.cc:1270] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250116 02:59:57.460683 23905 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.520263 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.520113 20682 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
I20250116 02:59:57.455586 23969 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.523011 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.522330 20682 raft_consensus.cc:604] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.524508 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.521581 20757 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:57.511222 23677 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:57.532817 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 202822us
I20250116 02:59:57.520881 23758 raft_consensus.cc:2798] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election won for term 8
I20250116 02:59:57.535362 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.533648 20572 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.537087 20750 raft_consensus.cc:2371] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5. Re-sending same reply.
I20250116 02:59:57.542232 23690 raft_consensus.cc:988] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:57.543903 23905 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.544600 23690 raft_consensus.cc:1076] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:57.544229 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.548168 23930 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.536165 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250116 02:59:57.552779 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 02:59:57.556787 20572 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:57.553638 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 02:59:57.559577 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.561262 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:57.563872 20608 raft_consensus.cc:1270] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250116 02:59:57.565106 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.521553 23973 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250116 02:59:57.566778 20675 raft_consensus.cc:2388] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 3: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:57.566936 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.566952 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:57.572412 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:57.568459 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:57.573817 21329 raft_consensus.cc:2388] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:57.569779 20572 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.572440 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:57.511027 23893 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.550448 23930 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.575198 20675 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:57.580040 20647 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.582660 23804 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:57.583256 23804 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.584100 23778 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
I20250116 02:59:57.584651 23778 raft_consensus.cc:695] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:57.584754 23936 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.585474 23804 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.585580 23778 consensus_queue.cc:237] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.586529 23965 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:57.586723 23874 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.563097 23905 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:57.588054 20647 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.590598 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:57.591301 20683 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:57.591430 20677 raft_consensus.cc:1235] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:57.593570 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.530074 23964 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.587478 23731 consensus_queue.cc:1035] T 31db17a68ff141f3bd799ccc59c81fcc P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20250116 02:59:57.595124 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.596735 20748 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 02:59:57.575788 23812 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250116 02:59:57.600138 23940 consensus_meta.cc:287] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.587s user 0.009s sys 0.003s
I20250116 02:59:57.600701 23940 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.570765 23973 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:57.602308 23973 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Advancing to term 10
I20250116 02:59:57.575531 23758 raft_consensus.cc:695] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:57.587491 23948 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.614943 20602 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43854
I20250116 02:59:57.614794 23758 consensus_queue.cc:237] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.615646 20602 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:59:57.616060 20602 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:57.601886 20751 raft_consensus.cc:1235] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:57.615136 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.621500 20754 raft_consensus.cc:2405] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20250116 02:59:57.622506 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 52835.7us
I20250116 02:59:57.604439 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:57.623350 20675 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:57.628607 20602 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.630201 20600 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:57.636826 20602 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.637372 21332 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 02:59:57.624312 21329 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 9.
I20250116 02:59:57.639344 20572 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:57.641858 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:57.645107 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:57.618752 23918 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.647490 23760 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:57.648074 23918 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.596962 23964 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:57.650595 20599 raft_consensus.cc:1235] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
W20250116 02:59:57.651521 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 77576.7us
I20250116 02:59:57.651607 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:57.637905 20720 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.650992 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.645994 21329 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
W20250116 02:59:57.657918 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 69946.9us
I20250116 02:59:57.652159 20683 raft_consensus.cc:2463] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 02:59:57.661180 20572 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:57.663158 21299 leader_election.cc:304] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:57.663729 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:57.664633 20675 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:57.664806 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.668941 21329 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:57.627173 23804 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:57.670094 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:57.675757 20682 raft_consensus.cc:3049] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:57.672840 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.640331 23905 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 02:59:57.676371 20682 raft_consensus.cc:738] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:57.648111 23760 raft_consensus.cc:695] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:57.691782 20571 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:57.685839 20647 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:57.660435 23948 raft_consensus.cc:2743] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:57.660902 23930 raft_consensus.cc:2798] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:57.640800 23956 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.650252 23677 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.715893 20683 raft_consensus.cc:1235] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
I20250116 02:59:57.670490 23683 raft_consensus.cc:2798] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election won for term 8
I20250116 02:59:57.709743 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.731202 20677 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:57.718367 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:57.733394 20720 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 02:59:57.734416 23964 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:57.734864 23964 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:57.735312 23964 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:57.735548 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.736667 20754 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:57.650146 23884 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20250116 02:59:57.737823 23804 raft_consensus.cc:2798] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:57.702507 23937 raft_consensus.cc:2798] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:57.738922 23937 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:57.739368 23937 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:57.722973 23930 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:57.719213 23918 consensus_queue.cc:1046] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250116 02:59:57.725005 23605 consensus_queue.cc:1046] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20250116 02:59:57.737461 23960 log.cc:926] Time spent T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20250116 02:59:57.711737 23915 consensus_meta.cc:287] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.777s user 0.012s sys 0.000s
I20250116 02:59:57.755285 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:57.756233 21329 raft_consensus.cc:2388] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 9: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 02:59:57.682351 20682 consensus_queue.cc:260] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.757249 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.738397 23804 raft_consensus.cc:695] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:57.742916 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.710244 23936 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:57.724994 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 63897.2us
I20250116 02:59:57.690685 23940 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.757546 20682 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:57.756669 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:57.709697 23944 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.748941 23930 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:57.756004 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:57.788921 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:57.759899 20571 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:57.747543 23893 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.732003 23589 consensus_queue.cc:1035] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20250116 02:59:57.752285 23683 raft_consensus.cc:695] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:57.762009 23760 consensus_queue.cc:237] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.803462 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.804471 20756 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 6
I20250116 02:59:57.766923 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.767243 23804 consensus_queue.cc:237] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.766985 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 02:59:57.770466 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 88950.5us
I20250116 02:59:57.771301 23605 raft_consensus.cc:3049] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:57.750996 23967 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:57.771927 23746 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.810804 20649 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.773622 23677 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.734773 23956 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.772130 23540 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.774784 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:57.775043 21300 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.783207 23905 raft_consensus.cc:3049] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:57.758353 20757 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250116 02:59:57.777433 23823 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.748736 23965 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20250116 02:59:57.707882 23969 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:57.782505 20572 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:57.787137 23944 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.797798 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:57.799105 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:57.807216 23605 raft_consensus.cc:738] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:57.815167 20675 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:57.775784 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:57.782876 23736 raft_consensus.cc:3049] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:57.830457 23736 raft_consensus.cc:738] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:57.820457 23758 consensus_queue.cc:1035] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250116 02:59:57.816933 23728 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:57.819989 23874 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 02:59:57.798930 23936 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:57.820770 23540 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.817730 23905 raft_consensus.cc:738] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:57.837967 20720 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.820744 23967 raft_consensus.cc:695] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:57.840518 20646 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.808010 23980 consensus_queue.cc:1046] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20250116 02:59:57.831521 23736 consensus_queue.cc:260] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.823710 23823 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.817368 23746 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:57.850853 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 68444.3us
I20250116 02:59:57.860263 20572 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:57.835062 23956 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:57.852646 20646 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.855077 21329 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 3.
I20250116 02:59:57.856475 23728 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:57.868958 23540 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.871742 23944 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:57.830729 23605 consensus_queue.cc:260] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.875552 23605 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:57.875844 23979 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:57.876833 23953 consensus_queue.cc:1159] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "17446c8dacf04461acb843547665250f" responder_term: 4 status { last_received { term: 4 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 4 index: 7 } }
I20250116 02:59:57.879195 23940 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.879756 23940 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.881974 23940 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:57.877256 23925 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election won for term 7
I20250116 02:59:57.883113 23925 raft_consensus.cc:695] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:57.884289 23925 consensus_queue.cc:237] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.887636 20572 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:57.888643 23914 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:57.876544 23979 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.890954 20723 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.895903 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:57.897584 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.902635 20757 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:57.903728 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.904470 20757 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 02:59:57.905383 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.899683 21327 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250116 02:59:57.906001 20757 raft_consensus.cc:3054] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:57.899466 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.845907 23969 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250116 02:59:57.894409 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 55212.2us
I20250116 02:59:57.905944 20675 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:57.846554 23936 raft_consensus.cc:695] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 02:59:57.912895 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 44665.4us
I20250116 02:59:57.914778 23728 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.918491 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.919301 20600 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:57.921773 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.832654 23984 consensus_queue.cc:1046] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20250116 02:59:57.923864 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.922725 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.889832 23903 raft_consensus.cc:2743] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250116 02:59:57.914705 23884 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:57.876158 23746 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.927035 21299 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:57.934852 20675 raft_consensus.cc:2388] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 02:59:57.936456 20752 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:57.975833 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 48939.1us
I20250116 02:59:57.938045 20647 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:57.926246 20755 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:57.943045 23984 raft_consensus.cc:3049] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:57.943840 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:57.944114 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.945477 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:57.947695 23954 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 02:59:57.845391 23683 consensus_queue.cc:237] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:57.935885 23992 log.cc:926] Time spent T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.085s user 0.002s sys 0.002s
W20250116 02:59:57.954360 23993 log.cc:926] Time spent T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20250116 02:59:57.956303 23956 raft_consensus.cc:2743] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 02:59:57.962976 23936 consensus_queue.cc:237] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:57.971218 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.973906 20757 raft_consensus.cc:2463] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:57.981446 21335 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:57.982909 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.984411 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.984661 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.989130 20680 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 3.
I20250116 02:59:57.922501 23940 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20250116 02:59:57.986579 23984 raft_consensus.cc:738] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:57.987102 20748 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:57.929634 23905 consensus_queue.cc:260] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:57.991196 20682 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 02:59:57.926134 23742 consensus_queue.cc:1035] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20250116 02:59:57.925802 23728 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:57.992511 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:58.014376 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:58.030516 21327 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 7.
I20250116 02:59:58.014791 20605 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:58.000998 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:58.004300 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:58.011242 20600 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:58.028625 20647 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:57.990953 20756 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:58.017311 20675 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20250116 02:59:57.939821 23932 consensus_queue.cc:1046] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20250116 02:59:58.020476 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:58.023936 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:58.019467 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:57.997031 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:58.016273 20607 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:58.031896 20720 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:57.992959 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:58.019191 21299 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:57.991158 20572 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:57.961911 23964 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:58.035041 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:57.997507 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:58.031788 20608 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:58.077363 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:58.030323 20757 raft_consensus.cc:2463] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
W20250116 02:59:58.079383 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 88448.2us
I20250116 02:59:58.080322 20601 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:58.080488 20572 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 02:59:58.082191 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 63114.8us
I20250116 02:59:58.027705 20752 raft_consensus.cc:2463] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 02:59:58.078719 20755 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 2.
I20250116 02:59:58.019712 23838 tablet_copy_client.cc:538] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250116 02:59:58.096364 23746 raft_consensus.cc:3049] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 02:59:58.096845 23746 raft_consensus.cc:738] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:58.097738 23746 consensus_queue.cc:260] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.098750 23914 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 02:59:58.100526 23728 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.993080 23802 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:58.001514 23967 consensus_queue.cc:237] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:58.101420 23802 raft_consensus.cc:695] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:58.107950 20571 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:58.109920 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:58.087296 20572 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:58.087025 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:58.111909 20600 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250116 02:59:58.089886 20608 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250116 02:59:58.090837 23559 raft_consensus.cc:988] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb: : Instructing follower f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:58.094597 21332 raft_consensus.cc:1270] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250116 02:59:58.094254 23937 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:58.088485 23605 raft_consensus.cc:2798] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:59:58.001066 23930 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:58.124315 23930 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:57.994676 23979 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:58.125483 20602 raft_consensus.cc:1235] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 02:59:58.001140 23933 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:57.987011 23580 raft_consensus.cc:2743] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250116 02:59:58.085618 23936 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:58.135861 23936 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:58.136399 23936 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:58.089015 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 57196.1us
I20250116 02:59:58.002189 23762 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:58.140695 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:58.079850 23997 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:58.129549 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:58.130838 23802 consensus_queue.cc:237] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.127061 23605 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:58.134980 23933 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.136091 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:58.340032 20756 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:58.111850 20607 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 02:59:58.022559 23940 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.097s
I20250116 02:59:58.125317 23937 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:58.115659 23948 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:57.996734 23973 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.141569 20752 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 02:59:58.118875 23804 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:58.324334 23561 raft_consensus.cc:1076] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Signalling peer f65e8d7f76144fab9725ca32a6328c3f to start an election
I20250116 02:59:58.126680 23964 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:58.341511 23605 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:58.356979 20604 raft_consensus.cc:1235] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
I20250116 02:59:58.345559 20649 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:58.100841 23812 consensus_queue.cc:1046] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.224s
W20250116 02:59:58.334213 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 340654us
I20250116 02:59:58.145128 23997 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:58.126870 23956 raft_consensus.cc:2743] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250116 02:59:58.391619 20677 raft_consensus.cc:2363] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 9. Current term is 10.
I20250116 02:59:58.351651 23948 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:58.356761 23965 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:58.369036 23804 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:58.330960 23884 consensus_queue.cc:1046] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20250116 02:59:58.402500 23948 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:58.397018 23997 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:58.410858 23804 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:58.139364 23984 consensus_queue.cc:260] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:58.467715 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 350410us
I20250116 02:59:58.471163 20723 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:58.473574 20723 leader_election.cc:304] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:58.443701 23678 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:58.485100 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:58.486595 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:58.499792 21330 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 LEADER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20250116 02:59:58.410568 23965 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.489454 23884 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 02:59:58.489619 23940 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:58.565204 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 159753us
I20250116 02:59:58.567451 24010 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:58.490810 23678 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 02:59:58.590442 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 196958us
W20250116 02:59:58.593618 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 120112us
I20250116 02:59:58.595327 20723 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:58.625413 20723 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:58.591673 23678 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:58.631482 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 244355us
I20250116 02:59:58.631667 20683 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:58.633330 20604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2"
dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:43854
I20250116 02:59:58.633975 20604 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:59:58.634373 20604 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:58.590662 23778 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:58.621841 23884 raft_consensus.cc:2743] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:58.690375 20608 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 02:59:58.699347 20604 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:58.716210 20604 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:58.722038 21333 raft_consensus.cc:3049] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Stepping down as leader of term 7
I20250116 02:59:58.722625 21333 raft_consensus.cc:738] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:58.592617 23940 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:58.726174 21333 consensus_queue.cc:260] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.694268 23778 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:58.736603 24020 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:58.724493 23884 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250116 02:59:58.796195 24020 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.690905 23559 consensus_queue.cc:1046] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20250116 02:59:58.799674 23559 raft_consensus.cc:3049] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Stepping down as leader of term 5
I20250116 02:59:58.800148 23559 raft_consensus.cc:738] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:58.801213 23559 consensus_queue.cc:260] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:58.743762 23965 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:58.690800 23933 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:58.745913 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:58.747493 24012 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:58.803432 24006 log.cc:926] Time spent T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20250116 02:59:58.749642 23973 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 10 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:58.805749 23965 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 02:59:58.819145 24017 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:58.806452 24012 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.820065 24017 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 02:59:58.820904 21331 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 7.
I20250116 02:59:58.822170 24012 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:58.825649 20599 consensus_meta.cc:287] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.740s user 0.011s sys 0.002s
I20250116 02:59:58.826241 20599 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
W20250116 02:59:58.825532 23999 log.cc:926] Time spent T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.715s user 0.002s sys 0.000s
I20250116 02:59:58.851559 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56674 (request call id 521) took 1065 ms. Trace:
I20250116 02:59:58.852231 21329 rpcz_store.cc:276] 0116 02:59:57.786413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.788678 (+ 2265us) service_pool.cc:225] Handling call
0116 02:59:58.851217 (+1062539us) spinlock_profiling.cc:243] Waited 65.4 ms on lock 0x7b54006f13d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:58.851446 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":150494080}
I20250116 02:59:58.597916 24010 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.783185 23778 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:58.775758 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 737) took 1848 ms. Trace:
I20250116 02:59:58.884006 20720 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:58.889537 20751 rpcz_store.cc:276] 0116 02:59:56.926951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:56.937177 (+ 10226us) service_pool.cc:225] Handling call
0116 02:59:56.937574 (+ 397us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:57.629017 (+691443us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:58.775433 (+1146416us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b5400397454. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 02:59:58.775689 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1527287680}
I20250116 02:59:58.896119 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:58.896451 20599 raft_consensus.cc:1270] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250116 02:59:58.816088 23838 tablet_bootstrap.cc:492] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap starting.
I20250116 02:59:58.896878 20604 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:58.795815 23940 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 02:59:58.898849 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 275824us
W20250116 02:59:58.893074 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 294381us
I20250116 02:59:58.879606 23559 raft_consensus.cc:3054] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:58.898757 23936 consensus_meta.cc:287] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.762s user 0.014s sys 0.000s
I20250116 02:59:58.905723 24020 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 02:59:58.918999 23605 consensus_meta.cc:287] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.519s user 0.004s sys 0.007s
I20250116 02:59:58.893777 23965 raft_consensus.cc:2798] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250116 02:59:58.929117 23605 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.932518 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 02:59:58.933451 21328 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:58.935935 20720 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:58.943940 21302 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:58.908280 20572 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:58.916370 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:58.901616 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 276309us
I20250116 02:59:58.946890 21336 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Advancing to term 10
I20250116 02:59:58.947847 20723 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:58.929563 23965 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:58.949703 20572 leader_election.cc:304] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 02:59:58.898165 23997 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:58.939292 24020 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 02:59:58.960157 20572 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:58.973531 21302 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 02:59:58.925969 23678 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:58.976665 20604 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:59.003576 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:59.004863 20681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1601) took 1447 ms (client timeout 1601 ms). Trace:
W20250116 02:59:59.005273 20681 rpcz_store.cc:269] 0116 02:59:57.557370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.557551 (+ 181us) service_pool.cc:225] Handling call
0116 02:59:59.003383 (+1445832us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b58001e0398. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 02:59:59.004761 (+ 1378us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6156544}
I20250116 02:59:58.950279 23930 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:58.926451 23936 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:59.009392 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 418380us
I20250116 02:59:58.960143 23965 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:59:59.009604 21336 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 10.
I20250116 02:59:59.009783 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2178) took 1070 ms. Trace:
I20250116 02:59:59.016233 20646 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:59.013731 20757 rpcz_store.cc:276] 0116 02:59:57.939506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:58.081543 (+142037us) service_pool.cc:225] Handling call
0116 02:59:58.082928 (+ 1385us) raft_consensus.cc:1466] Updating replica for 2 ops
0116 02:59:58.087946 (+ 5018us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:58.087979 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0116 02:59:58.090038 (+ 2059us) log.cc:844] Serialized 52 byte log entry
0116 02:59:58.505793 (+415755us) raft_consensus.cc:1640] Marking committed up to 3
0116 02:59:58.627323 (+121530us) log.cc:844] Serialized 14 byte log entry
0116 02:59:59.009397 (+382074us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:59.009485 (+ 88us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:59.009533 (+ 48us) raft_consensus.cc:1680] finished
0116 02:59:59.009547 (+ 14us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:59.009715 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":394469,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":183675,"raft.run_cpu_time_us":479,"raft.run_wall_time_us":17181,"thread_start_us":871027,"threads_started":3,"wal-append.queue_time_us":562491}
I20250116 02:59:58.961705 23956 raft_consensus.cc:2743] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:59:59.023401 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:59.024610 20681 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
W20250116 02:59:59.018719 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 117390us
I20250116 02:59:59.025139 20572 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:58.960310 24026 raft_consensus.cc:2798] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250116 02:59:58.981709 23996 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.007s sys 0.037s
W20250116 02:59:59.039163 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 65908.1us
I20250116 02:59:58.978930 24020 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.048760 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:59.010856 24007 raft_consensus.cc:2798] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:59.012781 23930 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:58.959136 23940 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20250116 02:59:59.062830 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:59.063489 20604 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Advancing to term 10
I20250116 02:59:59.064050 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:59.064754 20755 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 02:59:59.061921 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:59.065649 20682 raft_consensus.cc:2388] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 7: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:59.067864 20608 raft_consensus.cc:1235] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20250116 02:59:59.068043 24016 log.cc:926] Time spent T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20250116 02:59:59.040707 23760 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 10 FOLLOWER]: Leader election won for term 10
I20250116 02:59:59.072764 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:59.073652 20682 raft_consensus.cc:2388] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:59.019042 24029 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:59.026662 23914 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:59.051554 23996 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.935s user 0.007s sys 0.045s
I20250116 02:59:59.069056 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:59.039788 24032 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:59.079265 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:59.128360 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 103362us
I20250116 02:59:59.087301 20600 raft_consensus.cc:3049] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:59.089866 20720 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:59.098805 23605 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:59.069682 24007 raft_consensus.cc:695] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:59.101315 23997 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:59.102293 24017 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 02:59:59.060757 23930 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:59.121243 20571 leader_election.cc:304] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 02:59:59.125562 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:59.046583 24026 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:59.129168 20600 raft_consensus.cc:738] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 02:59:59.130805 20723 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 02:59:59.102764 23948 consensus_meta.cc:287] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.638s user 0.007s sys 0.005s
I20250116 02:59:59.092630 23936 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:59.050086 24020 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:59.139253 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:59.138437 20604 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 10.
W20250116 02:59:59.184939 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 54222.3us
I20250116 02:59:59.128551 24032 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:59.160671 20600 consensus_queue.cc:260] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:59.186189 20752 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 02:59:59.118405 23940 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:59.134760 24036 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:59.135041 24010 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 02:59:59.190533 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 80157.8us
I20250116 02:59:59.093391 23933 consensus_queue.cc:1046] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20250116 02:59:59.214203 20600 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:59.216341 23873 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:59.045347 24028 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.130s
I20250116 02:59:59.227479 23873 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.155653 23893 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:59.231173 23893 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.233676 23893 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:59.159915 24026 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 02:59:59.129338 23914 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.182852 24017 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:59.201802 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 02:59:59.237680 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 197415us
I20250116 02:59:59.206064 24032 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:59.252103 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:59.255347 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:59.255479 20600 raft_consensus.cc:1270] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20250116 02:59:59.262569 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 190103us
I20250116 02:59:59.185423 23948 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.261044 21328 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 8
I20250116 02:59:59.245942 23884 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 02:59:59.117651 24029 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:59.277747 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 121030us
I20250116 02:59:59.269392 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:59.278041 20647 leader_election.cc:336] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.767s (SENT)
I20250116 02:59:59.278648 20647 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:59.267859 23678 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:59.290294 20748 raft_consensus.cc:2363] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for earlier term 7. Current term is 8.
I20250116 02:59:59.291141 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 02:59:59.289686 23933 raft_consensus.cc:3049] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Stepping down as leader of term 6
I20250116 02:59:59.274374 23893 raft_consensus.cc:2743] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250116 02:59:59.219132 23940 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 02:59:59.277451 23760 raft_consensus.cc:695] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [term 10 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:59.258827 24017 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:59.275561 23873 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:59.259043 24007 consensus_queue.cc:237] T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:59.236178 24036 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:59.414911 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 171901us
I20250116 02:59:59.447364 20749 raft_consensus.cc:1235] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 4. Current term is 6. Ops: []
I20250116 02:59:59.316737 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:59.342440 20674 raft_consensus.cc:1270] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250116 02:59:59.309566 20678 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20250116 02:59:59.428154 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 139880us
I20250116 02:59:59.501699 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:59.509107 20679 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:59.510874 20607 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 02:59:59.433285 20681 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 02:59:59.511842 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:59.512641 20607 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
W20250116 02:59:59.447208 20750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50656 (request call id 2161) took 1989 ms (client timeout 1767 ms). Trace:
W20250116 02:59:59.514880 20750 rpcz_store.cc:269] 0116 02:59:57.458448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.458586 (+ 138us) service_pool.cc:225] Handling call
0116 02:59:59.446952 (+1988366us) spinlock_profiling.cc:243] Waited 75.9 ms on lock 0x7b5400043154. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:59.447125 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174680320}
I20250116 02:59:59.306974 20680 raft_consensus.cc:2388] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 02:59:59.511250 20647 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 02:59:59.517990 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:59.499464 20676 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
W20250116 02:59:59.425091 24002 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 1.070s user 0.000s sys 0.007s
I20250116 02:59:59.351696 24044 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:59.535413 24044 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 02:59:59.458884 23765 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20250116 02:59:59.346545 24043 log.cc:926] Time spent T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250116 02:59:59.527819 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:59.443559 24037 log.cc:926] Time spent T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.237s user 0.001s sys 0.001s
W20250116 02:59:59.317096 23804 consensus_meta.cc:287] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.841s user 0.012s sys 0.002s
I20250116 02:59:59.461570 21328 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:59.345443 24059 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.531867 24050 raft_consensus.cc:2798] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 02:59:59.532794 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 02:59:59.533655 23940 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:59.533823 23823 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 02:59:59.345928 24031 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:59.534933 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:59.288493 23884 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:59.453004 23998 log.cc:926] Time spent T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 1.339s user 0.013s sys 0.000s
I20250116 02:59:59.342350 24054 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:59.306883 24011 log.cc:926] Time spent T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20250116 02:59:59.295511 24024 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250116 02:59:59.548334 20604 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 02:59:59.443255 24053 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.442816 24048 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:59.445338 24046 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:59.403883 24035 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.350116 23997 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:59.456224 23948 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:59.561983 20720 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 02:59:59.503918 23933 raft_consensus.cc:738] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 02:59:59.346151 24045 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:59.577642 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 53907.9us
I20250116 02:59:59.345717 24033 consensus_queue.cc:1046] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.323s
I20250116 02:59:59.319569 24021 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:59.589215 20606 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 02:59:59.446352 23965 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.289788 24041 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.509s
I20250116 02:59:59.522532 23580 consensus_queue.cc:1035] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20250116 02:59:59.561146 23765 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.442867 24042 consensus_queue.cc:1035] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.183s
I20250116 02:59:59.583400 20600 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:59.557374 23760 consensus_queue.cc:237] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.583961 24050 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:59:59.724814 21332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 520) took 1938 ms. Trace:
I20250116 02:59:59.628983 20681 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250116 02:59:59.705361 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:59.758914 20678 raft_consensus.cc:3049] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
W20250116 02:59:59.750209 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 188311us
I20250116 02:59:59.761186 20750 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 6. Current term is 7. Ops: []
I20250116 02:59:59.762408 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:59.763008 20750 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 02:59:59.763115 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:59.763857 20752 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:59.591707 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 587) took 1608 ms. Trace:
I20250116 02:59:59.759572 20678 raft_consensus.cc:738] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 02:59:59.757900 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 02:59:59.780771 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 02:59:59.703804 24049 log.cc:926] Time spent T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.213s user 0.003s sys 0.000s
I20250116 02:59:59.769595 20603 rpcz_store.cc:276] 0116 02:59:57.983683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.983969 (+ 286us) service_pool.cc:225] Handling call
0116 02:59:57.984271 (+ 302us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:57.984661 (+ 390us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:57.984690 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:58.010413 (+ 25723us) log.cc:844] Serialized 26 byte log entry
0116 02:59:58.295739 (+285326us) raft_consensus.cc:1640] Marking committed up to 8
0116 02:59:58.296563 (+ 824us) log.cc:844] Serialized 14 byte log entry
0116 02:59:58.331092 (+ 34529us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:58.331191 (+ 99us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:59.353099 (+1021908us) spinlock_profiling.cc:243] Waited 7.09 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 02:59:59.587915 (+234816us) raft_consensus.cc:1680] finished
0116 02:59:59.591435 (+ 3520us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 02:59:59.591634 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34030,"prepare.queue_time_us":231,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":50,"raft.queue_time_us":44097,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":25732,"spinlock_wait_cycles":16301824,"thread_start_us":261184,"threads_started":1,"wal-append.queue_time_us":266123}
I20250116 02:59:59.802268 21302 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:59.756632 21332 rpcz_store.cc:276] 0116 02:59:57.786134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.786245 (+ 111us) service_pool.cc:225] Handling call
0116 02:59:58.094197 (+307952us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:58.112922 (+ 18725us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:59.724586 (+1611664us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54007a5b50. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 02:59:59.724742 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707526400}
I20250116 02:59:59.798580 20756 raft_consensus.cc:1270] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250116 02:59:59.771204 20678 consensus_queue.cc:260] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:59.805227 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:59.806381 20751 raft_consensus.cc:3054] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:59:59.806737 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:59.711797 24068 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.810230 24068 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.813020 24068 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:59.706418 24060 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.706636 24061 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.819502 24061 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.814121 24060 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.825794 24060 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:59.740551 23746 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 02:59:59.737416 23944 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.827158 20756 raft_consensus.cc:2388] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 02:59:59.841228 20723 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.013s sys 0.017s
W20250116 02:59:59.841656 20723 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.564s user 0.013s sys 0.017s
W20250116 02:59:59.841964 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 572462us
I20250116 02:59:59.577206 23804 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.699237 24058 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250116 02:59:59.779526 24002 log.cc:926] Time spent T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20250116 02:59:59.769192 24050 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 02:59:59.739095 24028 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 02:59:59.724210 23979 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.739995 24029 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:59.892092 20601 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20250116 02:59:59.735973 24017 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:59.842857 20723 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:59.892784 20750 raft_consensus.cc:2405] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20250116 02:59:59.893328 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 50578.3us
I20250116 02:59:59.650508 24031 raft_consensus.cc:695] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 02:59:59.799304 24045 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 02:59:59.805564 23930 consensus_meta.cc:287] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.616s user 0.013s sys 0.000s
I20250116 02:59:59.894024 20723 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 02:59:59.783541 24048 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:59.782989 23997 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.849951 20752 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 02:59:59.626405 23940 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 02:59:59.828398 24025 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.016s sys 0.000s
I20250116 02:59:59.828629 20600 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:59.916613 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:59.844357 24033 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:59:59.706663 24057 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 02:59:59.873232 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.021s sys 0.018s
W20250116 02:59:59.839843 24043 log.cc:926] Time spent T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250116 02:59:59.872839 20607 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 02:59:59.650861 23884 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:59.927421 23804 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:59.940999 23884 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:59.941668 23884 raft_consensus.cc:2798] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:59.845234 23948 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 02:59:59.781044 23765 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.848647 24061 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:59.851384 23823 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 02:59:59.909485 20751 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:59.913318 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 02:59:59.584441 24059 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.782868 24035 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.851310 23944 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.895257 24031 consensus_queue.cc:237] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:59.931682 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 02:59:59.932540 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.681s user 0.021s sys 0.023s
I20250116 02:59:59.783022 24046 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.896919 24028 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 02:59:59.934540 24073 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 02:59:59.935415 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 02:59:59.876713 24060 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 02:59:59.597401 24054 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:59.942028 23884 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:00.065179 23884 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:00.065457 23965 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 02:59:59.796272 24053 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.066511 20750 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:00.073537 20753 raft_consensus.cc:1270] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250116 03:00:00.073734 20604 raft_consensus.cc:2388] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:00.038517 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:00.075313 20608 raft_consensus.cc:2388] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 6: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:00.007606 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:00.076134 20600 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 8
W20250116 03:00:00.071743 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 819924us
I20250116 03:00:00.078372 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:00.079154 20756 raft_consensus.cc:3054] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 5
W20250116 02:59:59.929392 24025 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.017s sys 0.000s
I20250116 03:00:00.077790 21300 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 02:59:59.917207 24017 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 02:59:59.903918 24041 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 03:00:00.034596 23995 log.cc:926] Time spent T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 2.052s user 0.008s sys 0.026s
I20250116 02:59:59.974470 24036 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:59.900452 23930 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:00.023401 24045 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 02:59:59.923116 24070 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:00.119096 23765 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.071643 23948 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:00.107203 24073 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
W20250116 03:00:00.122287 20679 consensus_meta.cc:287] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.601s user 0.010s sys 0.003s
I20250116 02:59:59.941963 24044 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:00.110623 23965 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:00.133175 20756 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 03:00:00.127063 24036 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:00.137530 20750 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 8.
I20250116 02:59:59.922919 23940 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 02:59:59.897581 23979 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.142460 20679 raft_consensus.cc:2463] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
I20250116 02:59:59.928356 24056 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:00.115255 24017 raft_consensus.cc:695] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:00.147405 20599 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 03:00:00.148985 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:00.140868 23964 raft_consensus.cc:3049] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 7 LEADER]: Stepping down as leader of term 7
I20250116 03:00:00.079015 24060 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.134445 24054 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:00.156144 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 02:59:59.966476 24033 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.141912 23948 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:00.165700 23940 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:00.083889 23823 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.146018 24073 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:00.177824 24079 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:00.178833 20750 raft_consensus.cc:2363] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8.
I20250116 02:59:59.942482 24057 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 03:00:00.189601 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1836) took 2198 ms. Trace:
I20250116 03:00:00.190017 24017 consensus_queue.cc:237] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:00.190842 24020 consensus_meta.cc:287] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.983s user 0.013s sys 0.002s
I20250116 03:00:00.124743 24041 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.140719 24070 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:00.201452 24083 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:00.217926 20607 raft_consensus.cc:2388] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:00.219614 20749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1723) took 2695 ms. Trace:
I20250116 03:00:00.220134 20749 rpcz_store.cc:276] 0116 02:59:57.525990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.526129 (+ 139us) service_pool.cc:225] Handling call
0116 02:59:59.447124 (+1920995us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:59.518928 (+ 71804us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:00.219373 (+700445us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b5400043150. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ea09 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:00.219537 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4396450560}
I20250116 03:00:00.201673 23940 consensus_queue.cc:1035] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 03:00:00.202064 24066 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 03:00:00.022068 24081 consensus_queue.cc:1035] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.180s
I20250116 03:00:00.167888 23964 raft_consensus.cc:738] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:00.206877 20683 rpcz_store.cc:276] 0116 02:59:57.975438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.975554 (+ 116us) service_pool.cc:225] Handling call
0116 02:59:57.975811 (+ 257us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:58.632783 (+656972us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:00.173507 (+1540724us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b5400324954. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:00.173692 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1507789056}
I20250116 03:00:00.215588 20600 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 8.
I20250116 03:00:00.228045 24041 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.217233 24050 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:00.247994 20674 raft_consensus.cc:1235] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 7. Current term is 8. Ops: []
I20250116 03:00:00.248106 24033 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.218173 24020 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.249181 24050 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:00.249939 20755 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 03:00:00.250360 24020 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.175370 23997 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.157914 23965 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:00.253413 23965 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:00.156803 24036 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.205435 24077 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:00.208631 24059 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.211103 24048 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:00.235013 24028 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 03:00:00.247022 24032 consensus_meta.cc:287] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.978s user 0.011s sys 0.004s
I20250116 03:00:00.271515 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2175) took 2524 ms. Trace:
I20250116 03:00:00.245702 24066 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:00.272145 20754 rpcz_store.cc:276] 0116 02:59:57.746994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.747294 (+ 300us) service_pool.cc:225] Handling call
0116 02:59:57.760401 (+ 13107us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:57.780786 (+ 20385us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:57.780819 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:57.785246 (+ 4427us) log.cc:844] Serialized 26 byte log entry
0116 02:59:57.829886 (+ 44640us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:57.836115 (+ 6229us) log.cc:844] Serialized 14 byte log entry
0116 02:59:57.836341 (+ 226us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 02:59:57.836421 (+ 80us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 02:59:57.950934 (+114513us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b180035e1e0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd6914 00007fc323dd9cc1 00007fc323dd9c79 00007fc323dd9c09 00007fc323dd9bd1 00007fc323dd8ecd 00007fc32ce13234 00007fc32ce13069 00007fc323db8fe6
0116 03:00:00.012298 (+2061364us) spinlock_profiling.cc:243] Waited 6.32 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 03:00:00.271273 (+258975us) raft_consensus.cc:1680] finished
0116 03:00:00.271299 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:00.271444 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10212,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":6151,"raft.queue_time_us":199,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":2265,"spinlock_wait_cycles":19359872,"thread_start_us":44263,"threads_started":1,"wal-append.queue_time_us":140897}
I20250116 03:00:00.273468 24032 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.277082 24072 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:00.285414 24070 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:00.170202 24056 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.254964 24077 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:00.271625 23944 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 03:00:00.304769 20677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1830) took 2548 ms (client timeout 2113 ms). Trace:
W20250116 03:00:00.305335 20677 rpcz_store.cc:269] 0116 02:59:57.755913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.758053 (+ 2140us) service_pool.cc:225] Handling call
0116 03:00:00.304483 (+2546430us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b5400286254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:00.304666 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1294284032}
I20250116 03:00:00.279596 24066 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:00.276640 24053 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.283907 24046 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:00.292774 23930 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 03:00:00.337503 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 462328us
I20250116 03:00:00.231087 24079 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.361169 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 772) took 1109 ms. Trace:
I20250116 03:00:00.361652 20751 rpcz_store.cc:276] 0116 02:59:59.251666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.251828 (+ 162us) service_pool.cc:225] Handling call
0116 03:00:00.360950 (+1109122us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x7b540012d254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:00.361093 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1272516480}
I20250116 03:00:00.311079 24032 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:00.393216 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2450) took 2365 ms. Trace:
I20250116 03:00:00.327361 24036 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.394944 20602 rpcz_store.cc:276] 0116 02:59:57.997867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:58.086332 (+ 88465us) service_pool.cc:225] Handling call
0116 02:59:58.114218 (+ 27886us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:58.125997 (+ 11779us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:00.362986 (+2236989us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b54002918d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:00.363221 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25426176}
W20250116 03:00:00.363368 24026 consensus_meta.cc:287] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.107s user 0.012s sys 0.004s
I20250116 03:00:00.246469 24083 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.283501 23964 consensus_queue.cc:260] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.363193 24060 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.358103 24035 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.570827 24056 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.277841 24072 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.715344 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1803) took 1212 ms. Trace:
I20250116 03:00:00.652254 23964 raft_consensus.cc:3054] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
W20250116 03:00:00.748517 21301 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.878s user 0.107s sys 0.099s
W20250116 03:00:00.749389 21301 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.879s user 0.107s sys 0.099s
I20250116 03:00:00.750802 24032 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:00.751688 24032 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:00.754379 24032 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 03:00:00.755733 20572 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.006s sys 0.000s
W20250116 03:00:00.756237 20572 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.709s user 0.006s sys 0.000s
W20250116 03:00:00.756549 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 709528us
I20250116 03:00:00.757750 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:00.758790 21332 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:00.655823 23979 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:00.716136 20601 rpcz_store.cc:276] 0116 02:59:59.502542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.503137 (+ 595us) service_pool.cc:225] Handling call
0116 02:59:59.503544 (+ 407us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:59.892624 (+389080us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:00.714928 (+822304us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b54001f1dd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:00.715257 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":892838656}
W20250116 03:00:00.744063 20748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50646 (request call id 1733) took 2816 ms (client timeout 2150 ms). Trace:
I20250116 03:00:00.763185 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 03:00:00.763468 20748 rpcz_store.cc:269] 0116 02:59:57.903123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:58.030764 (+127641us) service_pool.cc:225] Handling call
0116 03:00:00.719349 (+2688585us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540012e654. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:00.719611 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2696351488}
W20250116 03:00:00.767092 20649 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.868s user 0.023s sys 0.042s
W20250116 03:00:00.767445 20649 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.873s user 0.023s sys 0.042s
W20250116 03:00:00.767714 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 896525us
W20250116 03:00:00.770913 20723 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.016s sys 0.023s
W20250116 03:00:00.771576 20723 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.764s user 0.016s sys 0.023s
I20250116 03:00:00.668691 24060 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:00.772837 21335 raft_consensus.cc:2363] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4.
I20250116 03:00:00.773835 20723 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 03:00:00.766202 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.963s user 0.024s sys 0.040s
W20250116 03:00:00.791239 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.988s user 0.024s sys 0.041s
W20250116 03:00:00.792270 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 988855us
I20250116 03:00:00.709059 23561 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:00.768182 21299 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.024s user 0.014s sys 0.014s
I20250116 03:00:00.828780 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:00.829156 21299 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.108s user 0.014s sys 0.014s
W20250116 03:00:00.829543 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 1.10877e+06us
I20250116 03:00:00.673759 24036 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:00.727885 20571 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.817s user 0.016s sys 0.037s
W20250116 03:00:00.835197 20571 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.936s user 0.017s sys 0.037s
W20250116 03:00:00.836134 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 937168us
W20250116 03:00:00.830461 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.009s sys 0.000s
W20250116 03:00:00.838722 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.689s user 0.010s sys 0.000s
I20250116 03:00:00.434558 24026 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:00.708992 23948 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:00.841145 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 69381.5us
I20250116 03:00:00.857316 24081 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:01.078537 23742 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:00.910816 23894 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:00.961045 23918 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:01.100695 20720 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.989s user 0.013s sys 0.005s
I20250116 03:00:00.937297 23981 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:00.832968 24060 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:01.078305 20648 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.013s sys 0.044s
I20250116 03:00:01.118019 21330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 524) took 3318 ms. Trace:
I20250116 03:00:01.099738 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:01.020324 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:00.841885 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 68107.7us
I20250116 03:00:00.761765 23565 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:01.124405 20648 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.245s user 0.013s sys 0.051s
I20250116 03:00:00.936534 21335 raft_consensus.cc:1270] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250116 03:00:00.901353 20647 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.022s user 0.009s sys 0.003s
W20250116 03:00:01.119203 20720 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.007s user 0.014s sys 0.005s
I20250116 03:00:01.116585 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:01.135974 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 1.02637e+06us
I20250116 03:00:01.126080 21330 rpcz_store.cc:276] 0116 02:59:57.791300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:57.791455 (+ 155us) service_pool.cc:225] Handling call
0116 02:59:57.791758 (+ 303us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:58.523625 (+731867us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:01.109306 (+2585681us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b5400790254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:01.109665 (+ 359us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1627400960}
I20250116 03:00:01.126473 21329 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 03:00:01.139194 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.140312 20683 raft_consensus.cc:2363] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8.
W20250116 03:00:01.144150 20647 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.264s user 0.009s sys 0.006s
I20250116 03:00:01.150305 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:01.151273 20751 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 4.
W20250116 03:00:01.117198 20722 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.015s sys 0.013s
W20250116 03:00:01.178470 20722 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.171s user 0.016s sys 0.013s
I20250116 03:00:01.183285 20649 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 03:00:01.110642 24007 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.974s user 0.027s sys 0.050s
W20250116 03:00:00.895552 23760 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.000s sys 0.009s
I20250116 03:00:00.787899 24057 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195)
I20250116 03:00:01.116756 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:01.079301 24070 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:01.258805 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 03:00:01.414384 20608 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.274s user 0.096s sys 0.182s
W20250116 03:00:01.483364 20608 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.343s user 0.096s sys 0.182s
W20250116 03:00:01.260730 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 77562.3us
I20250116 03:00:01.485904 20649 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:01.487018 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:01.446239 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:01.374248 20571 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 2.113s (SENT) [suppressed 7 similar messages]
I20250116 03:00:01.481076 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:01.489574 20757 raft_consensus.cc:1235] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 03:00:01.119072 24093 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:00.891356 23979 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader election won for term 8
W20250116 03:00:01.489562 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 115346us
I20250116 03:00:00.856491 23930 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:01.520534 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:01.078755 24077 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 03:00:01.380251 23884 consensus_meta.cc:287] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.314s user 0.007s sys 0.007s
I20250116 03:00:00.833684 24035 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:01.528203 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:01.117573 24079 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:01.118227 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:01.461951 24007 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.347s user 0.044s sys 0.062s
I20250116 03:00:01.379271 24087 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:01.379004 24095 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:01.380451 24083 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.381714 24060 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:01.382545 24102 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:01.481895 23760 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.244s user 0.000s sys 0.026s
I20250116 03:00:01.118708 23918 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.117949 23742 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.118361 23894 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.557911 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:01.591660 20754 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 03:00:01.119812 24021 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 03:00:01.532019 23731 raft_consensus.cc:2798] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election won for term 5
I20250116 03:00:01.524057 23776 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:01.578689 24007 raft_consensus.cc:491] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:01.517012 24070 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.393527 24091 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election won for term 8
W20250116 03:00:01.570652 21332 consensus_meta.cc:287] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.811s user 0.010s sys 0.001s
I20250116 03:00:01.571694 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:01.296164 24031 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:01.573266 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:00.857677 24036 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.545122 24077 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:01.637444 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:01.545367 23979 raft_consensus.cc:695] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 8 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
W20250116 03:00:01.585383 20572 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.018s sys 0.092s
I20250116 03:00:01.538794 23930 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.119626 23981 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.597857 24021 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:01.602540 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:00.842195 23561 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.547003 23884 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.574137 24096 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:01.657102 24096 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:01.659430 20680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1612) took 2608 ms (client timeout 1924 ms). Trace:
I20250116 03:00:01.659619 24096 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:01.569738 24017 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 03:00:01.659937 20680 rpcz_store.cc:269] 0116 02:59:59.051046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.051431 (+ 385us) service_pool.cc:225] Handling call
0116 03:00:01.659180 (+2607749us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54002fe254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:01.659341 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":420875136}
I20250116 03:00:01.660307 24017 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.663389 23980 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:01.663987 23980 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.665967 23937 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:01.666574 23937 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.667379 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1809) took 1764 ms. Trace:
I20250116 03:00:01.667763 20608 rpcz_store.cc:276] 0116 02:59:59.903435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:00.087955 (+184520us) service_pool.cc:225] Handling call
0116 03:00:00.114654 (+ 26699us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:00.115060 (+ 406us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:00.115104 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 03:00:00.124375 (+ 9271us) log.cc:844] Serialized 26 byte log entry
0116 03:00:01.483852 (+1359477us) raft_consensus.cc:1640] Marking committed up to 6
0116 03:00:01.483905 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:01.484008 (+ 103us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:01.667084 (+183076us) raft_consensus.cc:1680] finished
0116 03:00:01.667127 (+ 43us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:01.667312 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":629,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":5104,"thread_start_us":1343162,"threads_started":1,"wal-append.queue_time_us":1493253}
I20250116 03:00:01.669286 23884 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:01.117933 24081 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.616616 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:01.568167 24087 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.681031 21329 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 03:00:01.616775 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:01.620805 21332 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 8.
I20250116 03:00:01.625614 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:01.682256 21330 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 03:00:01.633466 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:01.634531 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:01.186813 23565 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.637524 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:01.573282 24095 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:01.643213 20572 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.018s sys 0.092s
I20250116 03:00:01.637578 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2470) took 2098 ms. Trace:
I20250116 03:00:01.615037 23731 raft_consensus.cc:695] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:01.607592 24092 raft_consensus.cc:491] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:01.694650 20606 rpcz_store.cc:276] 0116 02:59:59.539029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.780777 (+241748us) service_pool.cc:225] Handling call
0116 02:59:59.828108 (+ 47331us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:59.943328 (+115220us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:59.943362 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:59.943406 (+ 44us) raft_consensus.cc:1640] Marking committed up to 5
0116 02:59:59.943441 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:01.637231 (+1693790us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400254ed4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:01.637302 (+ 71us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:01.637495 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":264401664}
I20250116 03:00:01.644969 24091 raft_consensus.cc:695] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:01.589421 23933 consensus_queue.cc:260] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.638968 24086 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20250116 03:00:01.556043 24035 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:01.653167 20748 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.010s sys 0.019s
I20250116 03:00:01.560910 24093 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:01.744307 20748 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.623s user 0.010s sys 0.019s
I20250116 03:00:01.685885 21300 leader_election.cc:400] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647) with higher term. Message: Invalid argument: T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8.
I20250116 03:00:01.687005 20754 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:01.745735 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:01.745313 21300 leader_election.cc:403] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20250116 03:00:01.686520 20751 raft_consensus.cc:2388] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 03:00:01.746889 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 75660.1us
I20250116 03:00:01.662423 24017 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:01.586105 24102 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.613372 23776 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.623334 24007 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.639557 24031 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:01.733780 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 204807us
I20250116 03:00:01.695312 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:01.698824 23948 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.648278 24077 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.724794 23731 consensus_queue.cc:237] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.742066 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:01.651083 24066 consensus_meta.cc:287] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.326s user 0.012s sys 0.001s
W20250116 03:00:01.691830 20573 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.007s sys 0.033s
W20250116 03:00:01.766053 20573 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.762s user 0.008s sys 0.034s
I20250116 03:00:01.742286 24026 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 03:00:01.587322 21302 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.006s sys 0.033s
I20250116 03:00:01.749948 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.750366 21300 leader_election.cc:304] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:01.774715 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:01.778043 20680 raft_consensus.cc:2388] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 03:00:01.755264 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 149650us
I20250116 03:00:01.755539 21336 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 03:00:01.757218 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.781047 20677 raft_consensus.cc:2463] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:01.781456 20646 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:01.759869 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.747951 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:01.751603 24073 consensus_meta.cc:287] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.537s user 0.003s sys 0.013s
I20250116 03:00:01.736250 24106 raft_consensus.cc:491] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 03:00:01.782925 21302 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.702s user 0.006s sys 0.033s
I20250116 03:00:01.782784 20681 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 03:00:01.787488 20681 raft_consensus.cc:1235] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20250116 03:00:01.751081 24094 raft_consensus.cc:2743] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647) with higher term. Message: Invalid argument: T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8.
I20250116 03:00:01.710146 24092 raft_consensus.cc:513] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:01.794137 21299 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.007s sys 0.026s
I20250116 03:00:01.791169 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:01.799463 21299 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.674s user 0.007s sys 0.027s
W20250116 03:00:01.799927 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 688229us
I20250116 03:00:01.802426 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.809993 23905 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
W20250116 03:00:01.810747 24097 log.cc:926] Time spent T 27fcf38273074747a64f8e0c6d42de37 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20250116 03:00:01.811582 23776 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 03:00:01.812261 20574 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.038s sys 0.084s
W20250116 03:00:01.812649 20574 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.852s user 0.038s sys 0.084s
I20250116 03:00:01.813845 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.812700 23776 consensus_queue.cc:1046] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250116 03:00:01.815574 24102 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:01.816593 20680 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 03:00:01.817492 24035 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.819614 24007 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.820152 23936 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:01.820675 23936 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 03:00:01.821907 23905 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:01.822569 24093 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:01.822757 23905 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.824833 24031 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:01.768929 24108 consensus_queue.cc:1035] T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20250116 03:00:01.825091 23905 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:01.826689 23894 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.826968 24108 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:01.827459 23918 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:01.827617 24108 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.828356 23930 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:01.789592 24073 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.830543 24087 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:01.831478 24073 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:01.832484 23565 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:01.833377 23565 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 03:00:01.833248 20605 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 03:00:01.740425 23979 consensus_queue.cc:237] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.773782 24066 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.837702 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:01.841483 24036 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.841524 24109 raft_consensus.cc:2798] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election won for term 4
I20250116 03:00:01.823339 24093 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:01.844161 24109 raft_consensus.cc:695] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:01.841430 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.841856 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:01.842448 24105 log.cc:926] Time spent T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
I20250116 03:00:01.844653 24066 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:01.792570 23742 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:01.848026 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.852138 20682 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 03:00:01.801466 24070 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.808758 24057 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:01.833882 23565 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:01.847677 21328 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 03:00:01.860852 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:01.848639 24104 log.cc:926] Time spent T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20250116 03:00:01.831809 23561 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:01.844314 24093 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.834373 24091 consensus_queue.cc:237] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:01.898198 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 200714us
I20250116 03:00:01.791412 24106 raft_consensus.cc:513] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:01.858098 23981 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.863575 23565 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:01.918185 20646 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:01.910158 20572 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 03:00:01.912608 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 162332us
W20250116 03:00:01.914288 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 145931us
I20250116 03:00:01.921612 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:01.941190 21300 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:01.941354 20677 raft_consensus.cc:2363] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4.
I20250116 03:00:01.871611 24035 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 03:00:01.975364 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 65424.9us
I20250116 03:00:01.965291 24116 raft_consensus.cc:3049] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Stepping down as leader of term 3
I20250116 03:00:01.849586 24109 consensus_queue.cc:237] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.969419 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.920459 24081 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.973575 24108 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:01.979151 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:01.979754 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:01.867491 24057 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.981307 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:01.966861 24107 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:01.964340 23731 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 03:00:01.996454 20601 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 03:00:02.020704 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:02.004609 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:02.005462 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:02.022718 20607 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 8: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:02.002357 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:02.024147 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:02.024951 20604 raft_consensus.cc:2388] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 8: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 03:00:02.013036 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:02.012319 20723 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.000s sys 0.023s
I20250116 03:00:02.026650 24005 raft_consensus.cc:3049] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Stepping down as leader of term 5
I20250116 03:00:02.027822 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
W20250116 03:00:02.153689 20723 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.692s user 0.000s sys 0.024s
I20250116 03:00:02.031008 24031 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.019511 24026 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:02.019707 24102 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:02.031854 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:02.034267 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.991353 24035 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.991879 23981 raft_consensus.cc:2798] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader election won for term 4
I20250116 03:00:01.966125 24095 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.031386 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:01.985431 24116 raft_consensus.cc:738] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
W20250116 03:00:02.053155 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 69890.9us
I20250116 03:00:02.056613 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:01.990202 24070 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:01.995366 24087 raft_consensus.cc:491] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 03:00:02.016422 23964 consensus_meta.cc:287] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 1.238s user 0.013s sys 0.002s
I20250116 03:00:02.072779 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:02.073098 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:01.913267 24021 raft_consensus.cc:513] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:01.995501 24115 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.004801 24110 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 03:00:02.005482 24106 leader_election.cc:290] T 5ba18c5d216643cbb5689397ab22da23 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:02.019496 23731 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:02.077953 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:02.021322 24081 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:02.012341 24114 raft_consensus.cc:491] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:02.075799 23894 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.334954 23894 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.337536 23894 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.075325 23930 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.077775 23905 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20250116 03:00:02.337978 23823 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.338165 24081 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.119192 24092 leader_election.cc:290] T 1d7658a3a42c44f485085e75a4bd7b88 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.053176 24107 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 03:00:02.340279 24081 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.341270 24107 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:02.105931 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.341763 23937 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.110821 24051 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.343304 24051 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.289163 24070 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.341812 24114 raft_consensus.cc:513] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.345420 24051 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.131875 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:02.136857 20755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 764) took 3012 ms (client timeout 1615 ms). Trace:
I20250116 03:00:02.123972 24111 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.152238 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 782) took 1181 ms. Trace:
I20250116 03:00:02.153877 20754 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
W20250116 03:00:02.157512 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 158806us
W20250116 03:00:02.154258 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 693115us
I20250116 03:00:02.350577 20748 rpcz_store.cc:276] 0116 03:00:00.970794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:00.971003 (+ 209us) service_pool.cc:225] Handling call
0116 03:00:01.066447 (+ 95444us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:01.066765 (+ 318us) raft_consensus.cc:1529] Early marking committed up to index 8
0116 03:00:01.744731 (+677966us) log.cc:844] Serialized 14 byte log entry
0116 03:00:02.143481 (+398750us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:02.143682 (+ 201us) raft_consensus.cc:1640] Marking committed up to 8
0116 03:00:02.143750 (+ 68us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:02.151882 (+ 8132us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:02.152135 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":765304,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":2277,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":45504,"thread_start_us":1021332,"threads_started":2}
I20250116 03:00:02.125310 24136 consensus_queue.cc:1035] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.120s
I20250116 03:00:02.234145 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:02.248046 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 53532.5us
I20250116 03:00:02.191491 24126 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.192246 24063 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.183985 24005 raft_consensus.cc:738] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:02.193770 24102 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.391832 24102 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.184543 24052 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.392693 24052 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.311543 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:02.294459 20677 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:02.302176 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:02.292753 20608 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:02.241292 24035 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.191928 24026 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.302023 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.302213 20756 raft_consensus.cc:1270] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20250116 03:00:02.411736 20756 pending_rounds.cc:77] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f: Aborting all ops after (but not including) 8
I20250116 03:00:02.412068 20756 pending_rounds.cc:99] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20250116 03:00:02.412329 20756 raft_consensus.cc:2883] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 03:00:02.321282 21330 raft_consensus.cc:1235] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20250116 03:00:02.323016 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.239856 23981 raft_consensus.cc:695] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 03:00:02.294266 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 49326us
I20250116 03:00:02.326184 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:02.289372 24077 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:02.332162 23731 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:02.190315 24031 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:02.346635 20755 rpcz_store.cc:269] 0116 02:59:59.124874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.125076 (+ 202us) service_pool.cc:225] Handling call
0116 03:00:02.136665 (+3011589us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540012d754. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:02.136780 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2522753792}
I20250116 03:00:02.355654 24114 leader_election.cc:290] T 8d8e1b8edadd42f08808ffaf11b40c8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.233605 24116 consensus_queue.cc:260] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:02.302327 24087 raft_consensus.cc:513] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.360363 24111 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.360738 23565 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.361147 23930 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:02.395074 24063 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.325373 24057 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.328239 24110 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:02.362432 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.363162 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 02:59:58.724695 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 03:00:02.367622 20681 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:02.358100 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.378024 24126 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.349877 24070 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.396687 20679 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 4.
I20250116 03:00:02.392750 24021 leader_election.cc:290] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:02.396646 20607 raft_consensus.cc:2463] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:02.452244 24052 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.403307 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:02.387620 23964 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.349141 24115 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:02.414482 21332 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 03:00:02.464901 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:02.465279 20720 leader_election.cc:400] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4.
I20250116 03:00:02.414870 20602 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:02.465597 20720 leader_election.cc:403] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250116 03:00:02.465461 20646 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:02.422782 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.465502 21332 raft_consensus.cc:3054] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:02.430086 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.417644 20571 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 03:00:02.429644 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:02.467149 21330 raft_consensus.cc:2388] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 03:00:02.413635 24102 raft_consensus.cc:491] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.454820 21336 raft_consensus.cc:2363] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8.
I20250116 03:00:02.438666 20682 raft_consensus.cc:2363] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6.
I20250116 03:00:02.440280 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2183) took 3403 ms. Trace:
I20250116 03:00:02.444236 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:02.468914 20681 raft_consensus.cc:2463] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
W20250116 03:00:02.471447 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 53925.1us
I20250116 03:00:02.414125 20608 raft_consensus.cc:2463] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
I20250116 03:00:02.472029 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:02.461053 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:02.468569 20757 rpcz_store.cc:276] 0116 02:59:59.036832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.036997 (+ 165us) service_pool.cc:225] Handling call
0116 02:59:59.037399 (+ 402us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:01.495387 (+2457988us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:02.439942 (+944555us) spinlock_profiling.cc:243] Waited 2.45 s on lock 0x7b540012e154. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:02.440196 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5638891264}
I20250116 03:00:02.474309 24131 raft_consensus.cc:2798] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 03:00:02.477007 20571 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:02.447150 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:02.443542 21335 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 8: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 03:00:02.444640 21329 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:02.450506 24126 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.474766 23760 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 03:00:02.472898 20748 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 03:00:02.478160 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.477921 20679 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:02.479601 21332 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 03:00:02.474265 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:02.486455 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:02.495569 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.495534 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:02.467161 23925 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 4.026s user 0.012s sys 0.042s
I20250116 03:00:02.496352 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.509392 20646 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:02.437842 23565 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.509138 20755 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 03:00:02.438354 23984 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:02.512552 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.515292 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.516607 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:02.440254 23930 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:02.474395 23914 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.442159 24110 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:02.469797 24063 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.471700 23937 raft_consensus.cc:2743] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 03:00:02.521618 21335 raft_consensus.cc:2463] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 7.
I20250116 03:00:02.526298 21330 raft_consensus.cc:2388] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 8: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 03:00:02.526485 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:02.524230 21332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.522745 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:02.474844 24131 raft_consensus.cc:491] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 03:00:02.509299 23925 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 4.068s user 0.012s sys 0.042s
I20250116 03:00:02.539008 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:02.486562 24107 raft_consensus.cc:2798] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 03:00:02.486562 23760 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:02.515268 23874 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:02.434405 24119 consensus_queue.cc:1046] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.101s
I20250116 03:00:02.621526 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:02.541731 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:02.623278 20755 raft_consensus.cc:2405] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250116 03:00:02.551698 24124 consensus_queue.cc:1035] T 49cb7aba83e046129200b93f32eb3585 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.073s
I20250116 03:00:02.540125 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:02.475505 24102 raft_consensus.cc:513] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.545189 24031 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:02.471143 23964 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:02.622323 20757 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 03:00:02.628157 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:02.545848 24026 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:02.554486 20748 raft_consensus.cc:2463] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 03:00:02.521545 24012 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:02.515444 24023 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:02.534716 23979 raft_consensus.cc:2743] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 3. Current term is 4.
I20250116 03:00:02.522122 24126 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election won for term 6
I20250116 03:00:02.586946 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2474) took 2701 ms. Trace:
I20250116 03:00:02.760911 20603 rpcz_store.cc:276] 0116 02:59:59.885692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.885949 (+ 257us) service_pool.cc:225] Handling call
0116 02:59:59.886322 (+ 373us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:00.288258 (+401936us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:00.288298 (+ 40us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:00.288351 (+ 53us) raft_consensus.cc:1640] Marking committed up to 9
0116 03:00:00.288387 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:02.586639 (+2298252us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b5400290ed4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:02.586694 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:02.586871 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":923741056}
I20250116 03:00:02.628793 20757 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 03:00:02.460177 24005 consensus_queue.cc:260] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:02.868147 24023 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:02.756387 20648 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.506s user 0.009s sys 0.011s
I20250116 03:00:03.169070 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:03.232641 20602 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:03.233906 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:03.046818 20607 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.000s sys 0.023s
W20250116 03:00:03.235292 20607 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.766s user 0.002s sys 0.026s
I20250116 03:00:02.578562 24131 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Advancing to term 9
W20250116 03:00:02.957950 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 417918us
I20250116 03:00:03.234751 20602 raft_consensus.cc:2463] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:03.249660 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 03:00:03.039659 20608 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.010s sys 0.013s
I20250116 03:00:03.249912 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:03.250437 20608 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s user 0.012s sys 0.015s
I20250116 03:00:02.762529 23823 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250116 03:00:03.258064 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:03.260869 24023 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
W20250116 03:00:03.260653 20754 consensus_meta.cc:287] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.910s user 0.007s sys 0.004s
I20250116 03:00:02.579484 24107 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 03:00:03.098663 24145 log.cc:926] Time spent T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20250116 03:00:02.763985 24012 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:03.075464 24125 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.523s user 0.001s sys 0.000s
W20250116 03:00:03.079725 24141 log.cc:926] Time spent T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.205s user 0.000s sys 0.001s
W20250116 03:00:03.100703 24147 log.cc:926] Time spent T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.172s user 0.000s sys 0.002s
I20250116 03:00:03.031491 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2476) took 1053 ms. Trace:
I20250116 03:00:02.865173 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 03:00:03.183982 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 419661us
I20250116 03:00:02.637354 23874 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:02.963183 21300 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 03:00:02.927141 24129 raft_consensus.cc:491] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:03.369088 20606 rpcz_store.cc:276] 0116 03:00:01.977891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:01.978094 (+ 203us) service_pool.cc:225] Handling call
0116 03:00:01.985008 (+ 6914us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:02.013382 (+ 28374us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:02.013415 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 03:00:02.015957 (+ 2542us) log.cc:844] Serialized 26 byte log entry
0116 03:00:02.300708 (+284751us) raft_consensus.cc:1640] Marking committed up to 3
0116 03:00:02.300750 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:02.300828 (+ 78us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:02.698640 (+397812us) raft_consensus.cc:1680] finished
0116 03:00:02.698682 (+ 42us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:03.031396 (+332714us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28362,"raft.queue_time_us":28967,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":52427,"thread_start_us":214741,"threads_started":1,"wal-append.queue_time_us":392606}
I20250116 03:00:02.701997 24111 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:02.551005 23981 consensus_queue.cc:237] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:02.755440 23937 raft_consensus.cc:491] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:02.755439 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 290303us
I20250116 03:00:03.236846 20679 raft_consensus.cc:1270] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250116 03:00:03.199882 24087 leader_election.cc:290] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:03.261153 23823 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 03:00:03.571398 23823 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:03.571905 23823 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:02.896219 24126 raft_consensus.cc:695] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Leader. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:03.575362 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.576078 23874 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:03.576527 20679 raft_consensus.cc:2405] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250116 03:00:03.577468 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1624) took 1550 ms. Trace:
I20250116 03:00:03.577913 20682 rpcz_store.cc:276] 0116 03:00:02.027458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.065896 (+ 38438us) service_pool.cc:225] Handling call
0116 03:00:03.577223 (+1511327us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54002fe254. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:03.577397 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":245849472}
I20250116 03:00:02.589099 23760 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:03.263558 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:03.263926 20754 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
W20250116 03:00:03.582522 20754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 790) took 1560 ms (client timeout 1841 ms). Trace:
W20250116 03:00:03.582949 20754 rpcz_store.cc:269] 0116 03:00:02.021853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.027542 (+ 5689us) service_pool.cc:225] Handling call
0116 03:00:03.582441 (+1554899us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 03:00:03.266289 20676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1843) took 3715 ms. Trace:
I20250116 03:00:03.583990 20676 rpcz_store.cc:276] 0116 02:59:59.555476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.556521 (+ 1045us) service_pool.cc:225] Handling call
0116 02:59:59.556974 (+ 453us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 02:59:59.966814 (+409840us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:59.966849 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 02:59:59.966895 (+ 46us) raft_consensus.cc:1640] Marking committed up to 9
0116 02:59:59.966935 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.266034 (+3299099us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54001928d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:03.266088 (+ 54us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:03.266224 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":941917568}
I20250116 03:00:03.584709 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.585737 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.252825 24005 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:03.586627 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:03.378298 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.392670 21329 raft_consensus.cc:1270] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20250116 03:00:03.267345 24107 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 03:00:03.581878 20602 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:03.586484 20676 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 9.
I20250116 03:00:03.589833 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.557370 20748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 796) took 1122 ms. Trace:
I20250116 03:00:03.378950 24070 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:03.558527 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 133637us
I20250116 03:00:03.446038 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:03.443579 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 480418us
I20250116 03:00:03.344192 24148 raft_consensus.cc:491] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 03:00:03.378465 23984 consensus_meta.cc:287] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.853s user 0.006s sys 0.004s
I20250116 03:00:03.563424 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:03.557222 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 148313us
I20250116 03:00:03.370373 23690 raft_consensus.cc:491] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:03.563830 20755 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 03:00:03.567099 20720 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:03.607942 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1812) took 1583 ms. Trace:
I20250116 03:00:03.609555 20720 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 03:00:03.486037 23838 log.cc:926] Time spent T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20250116 03:00:03.399603 24129 raft_consensus.cc:513] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:03.456892 23605 raft_consensus.cc:2758] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250116 03:00:03.361217 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56674 (request call id 529) took 4671 ms. Trace:
I20250116 03:00:03.587417 20679 raft_consensus.cc:3054] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:03.590852 20748 rpcz_store.cc:276] 0116 03:00:02.462487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:03.395436 (+932949us) service_pool.cc:225] Handling call
0116 03:00:03.468157 (+ 72721us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:03.557014 (+ 88857us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:03.557039 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:03.557076 (+ 37us) raft_consensus.cc:1640] Marking committed up to 8
0116 03:00:03.557110 (+ 34us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.557168 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:03.557315 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 03:00:03.599779 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.603483 24012 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:03.603854 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.622282 20754 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 03:00:03.604182 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:03.623728 21335 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
W20250116 03:00:03.606390 23731 consensus_meta.cc:287] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.177s user 0.010s sys 0.000s
I20250116 03:00:03.606714 23957 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:03.606868 21334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56674 (request call id 540) took 1988 ms (client timeout 2415 ms). Trace:
I20250116 03:00:03.608052 21302 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 03:00:03.618825 21327 rpcz_store.cc:276] 0116 02:59:58.690245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:58.691541 (+ 1296us) service_pool.cc:225] Handling call
0116 02:59:58.700807 (+ 9266us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 02:59:58.902127 (+201320us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 02:59:58.902168 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 02:59:58.909173 (+ 7005us) log.cc:844] Serialized 26 byte log entry
0116 02:59:59.013921 (+104748us) raft_consensus.cc:1640] Marking committed up to 6
0116 02:59:59.013964 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.265747 (+4251783us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54004907d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:03.265808 (+ 61us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:03.265858 (+ 50us) raft_consensus.cc:1680] finished
0116 03:00:03.265869 (+ 11us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:03.361095 (+ 95226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21263,"raft.queue_time_us":98,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":3042,"spinlock_wait_cycles":462012160,"thread_start_us":84923,"threads_started":1,"wal-append.queue_time_us":325387,"wal-append.run_cpu_time_us":2756,"wal-append.run_wall_time_us":1081682}
I20250116 03:00:03.614434 20720 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:03.627226 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.629455 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:03.601521 23600 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:03.594310 24115 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:03.632958 20720 leader_election.cc:400] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8.
W20250116 03:00:03.626601 21334 rpcz_store.cc:269] 0116 03:00:01.618498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:01.686921 (+ 68423us) service_pool.cc:225] Handling call
0116 03:00:03.563186 (+1876265us) spinlock_profiling.cc:243] Waited 8.4 ms on lock 0x7b5800199898. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323d98991 00007fc323d98842 00007fc32ca69cd9 00007fc32bbf8770 00007fc32bcbe81b 00007fc32bcbe8a1 00007fc32bc6e1c0 00007fc32bc6daa4 00007fc32533bc14 00007fc32533bba2
0116 03:00:03.606805 (+ 43619us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19318656}
I20250116 03:00:03.633347 20720 leader_election.cc:403] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250116 03:00:03.633885 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:03.631338 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:03.634207 20679 raft_consensus.cc:2463] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 03:00:03.635257 21335 raft_consensus.cc:2388] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 03:00:03.634732 21327 raft_consensus.cc:2463] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 03:00:03.636081 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 03:00:03.631932 24095 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.016s sys 0.024s
I20250116 03:00:03.636859 20748 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 03:00:03.618333 24119 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 03:00:03.642917 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:03.640545 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:03.641075 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:03.645401 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:03.588068 23937 raft_consensus.cc:513] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:03.650766 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:03.653738 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:03.622946 24122 raft_consensus.cc:2798] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 03:00:03.606441 24148 raft_consensus.cc:513] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:03.657784 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:03.658577 20754 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 3.
I20250116 03:00:03.659952 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:03.575469 24126 consensus_queue.cc:237] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:03.660842 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb87db38c5904a70bb4b4b6d11afa1ca" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:03.661177 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:03.633095 24087 raft_consensus.cc:2798] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 03:00:03.632632 23957 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:03.671722 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:03.639163 24152 raft_consensus.cc:2743] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959) with higher term. Message: Invalid argument: T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8.
I20250116 03:00:03.600957 24070 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:03.638964 24150 consensus_queue.cc:1046] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20250116 03:00:03.649726 24119 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:03.683123 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:03.617615 23690 raft_consensus.cc:513] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:03.692652 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 75728.9us
I20250116 03:00:03.695092 21300 leader_election.cc:304] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:03.698393 24163 raft_consensus.cc:2798] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election won for term 8
I20250116 03:00:03.673655 24023 raft_consensus.cc:2798] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 03:00:03.665293 24122 raft_consensus.cc:491] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:03.629431 23731 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:03.699064 24163 raft_consensus.cc:695] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:03.693868 24119 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 03:00:03.689990 24087 raft_consensus.cc:695] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:03.643952 23600 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:03.808945 20572 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.014s sys 0.003s
W20250116 03:00:03.835907 20572 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.586s user 0.015s sys 0.004s
W20250116 03:00:03.836654 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 587502us
I20250116 03:00:03.609031 20608 rpcz_store.cc:276] 0116 03:00:02.024511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.515193 (+490682us) service_pool.cc:225] Handling call
0116 03:00:02.515788 (+ 595us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:02.525797 (+ 10009us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:02.525830 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 03:00:02.526336 (+ 506us) log.cc:844] Serialized 26 byte log entry
0116 03:00:03.251002 (+724666us) raft_consensus.cc:1640] Marking committed up to 8
0116 03:00:03.505558 (+254556us) log.cc:844] Serialized 14 byte log entry
0116 03:00:03.558655 (+ 53097us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.558743 (+ 88us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:03.607675 (+ 48932us) raft_consensus.cc:1680] finished
0116 03:00:03.607709 (+ 34us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:03.607873 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71291,"prepare.queue_time_us":305426,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":11069,"raft.run_cpu_time_us":471,"raft.run_wall_time_us":31163,"thread_start_us":945727,"threads_started":2,"wal-append.queue_time_us":1029654}
I20250116 03:00:03.839639 21334 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:03.840746 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:03.624918 20647 thread.cc:635] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.014s sys 0.015s
W20250116 03:00:03.844499 20647 thread.cc:601] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.749s user 0.014s sys 0.015s
W20250116 03:00:03.845252 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 749780us
I20250116 03:00:03.598193 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:03.847710 23731 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:03.630836 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:03.557145 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 78304.4us
I20250116 03:00:03.816951 20680 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:03.817981 20683 raft_consensus.cc:2363] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6.
I20250116 03:00:03.814113 20677 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:03.818733 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:03.820541 20682 raft_consensus.cc:1270] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 03:00:03.826177 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:03.826925 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.727553 23838 tablet_bootstrap.cc:492] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20250116 03:00:03.889402 20749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35540 (request call id 786) took 2759 ms (client timeout 2975 ms). Trace:
I20250116 03:00:03.748375 23937 leader_election.cc:290] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:03.754246 23980 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:03.753028 24023 raft_consensus.cc:695] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:03.728785 24070 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:03.744406 24087 consensus_queue.cc:237] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:03.768949 24072 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:03.718044 24093 leader_election.cc:290] T 49f9e38171174b70b4373b56dbcd90be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 03:00:03.731031 24101 log.cc:926] Time spent T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 2.360s user 0.023s sys 0.015s
I20250116 03:00:03.733413 24102 leader_election.cc:290] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:03.757944 24107 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:03.967861 24107 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 03:00:03.747256 24110 consensus_meta.cc:287] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 1.193s user 0.012s sys 0.000s
I20250116 03:00:03.968675 23736 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 03:00:03.969838 24110 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:03.972723 20649 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:03.973253 24110 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:03.705731 24122 raft_consensus.cc:3054] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 03:00:03.728780 24129 leader_election.cc:290] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:03.978088 24005 raft_consensus.cc:491] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
W20250116 03:00:03.814818 20607 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.531s user 0.006s sys 0.016s
I20250116 03:00:03.746834 24148 leader_election.cc:290] T a569f56f9d2143869b29c78321047d00 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:03.749847 24163 consensus_queue.cc:237] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:03.752321 24162 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20250116 03:00:04.010937 24005 raft_consensus.cc:513] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:03.769301 24099 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.115s user 0.011s sys 0.014s
W20250116 03:00:04.015350 24099 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.365s user 0.011s sys 0.015s
I20250116 03:00:03.773057 24119 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:04.019173 24119 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:03.813910 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.020606 20606 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 03:00:04.022078 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:03.849777 20676 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 03:00:03.860319 20754 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
W20250116 03:00:03.860711 20605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1804) took 4352 ms (client timeout 2116 ms). Trace:
I20250116 03:00:03.863019 20599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43854 (request call id 2461) took 4817 ms. Trace:
I20250116 03:00:03.865523 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:03.850131 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:03.862283 23823 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:03.867518 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 598) took 1811 ms. Trace:
I20250116 03:00:03.868569 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:03.885834 23802 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:03.887041 23600 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:03.887259 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2191) took 3793 ms. Trace:
I20250116 03:00:03.887843 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50656 (request call id 2188) took 3952 ms. Trace:
I20250116 03:00:03.880007 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.876019 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34866 (request call id 1844) took 4308 ms. Trace:
I20250116 03:00:03.872694 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:03.864778 20504 catalog_manager.cc:5526] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 03:00:03.905119 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 1071) took 1540 ms. Trace:
I20250116 03:00:03.911171 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f82fb4c6a24889be1c1e2fe3e7d3e5" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:03.839943 23690 leader_election.cc:290] T c881ef7597c74adfbc9e875bfb6e5f7f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:04.033994 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:03.914212 21333 raft_consensus.cc:3054] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:04.028584 20753 rpcz_store.cc:276] 0116 03:00:00.094829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:00.098759 (+ 3930us) service_pool.cc:225] Handling call
0116 03:00:00.114464 (+ 15705us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:00.252408 (+137944us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:00.252446 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:00.252493 (+ 47us) raft_consensus.cc:1640] Marking committed up to 2
0116 03:00:00.252526 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.886508 (+3633982us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b5400044a54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:03.886600 (+ 92us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:03.886986 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316558592}
I20250116 03:00:04.035569 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.036317 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:04.036985 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.037361 20753 raft_consensus.cc:2388] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 8: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 03:00:03.930955 20572 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 03:00:03.932144 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:03.926831 23838 tablet_bootstrap.cc:492] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Bootstrap complete.
I20250116 03:00:04.039289 20608 raft_consensus.cc:2405] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20250116 03:00:04.040141 23838 ts_tablet_manager.cc:1397] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent bootstrapping tablet: real 5.224s user 0.059s sys 0.047s
I20250116 03:00:04.040463 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:03.889946 20749 rpcz_store.cc:269] 0116 03:00:01.129654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:01.129775 (+ 121us) service_pool.cc:225] Handling call
0116 03:00:03.888797 (+2759022us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540012e654. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:03.889139 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2361981312}
I20250116 03:00:03.937302 24023 consensus_queue.cc:237] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:03.940567 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 312911us
W20250116 03:00:03.941687 20680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1618) took 2100 ms (client timeout 1734 ms). Trace:
W20250116 03:00:03.955509 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 258697us
I20250116 03:00:03.958065 23580 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:03.775700 24116 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 03:00:04.066123 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:03.975488 24120 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 03:00:03.976966 24088 raft_consensus.cc:491] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:04.070881 24088 raft_consensus.cc:513] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:03.737946 24131 raft_consensus.cc:513] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:03.972752 24149 log.cc:926] Time spent T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.372s user 0.001s sys 0.002s
I20250116 03:00:04.086248 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.087679 20602 raft_consensus.cc:2388] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 9: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:04.088028 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53964 (request call id 603) took 2085 ms. Trace:
I20250116 03:00:04.032585 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:04.032907 23823 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:04.024603 20599 rpcz_store.cc:276] 0116 02:59:59.045422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.045579 (+ 157us) service_pool.cc:225] Handling call
0116 02:59:59.068070 (+ 22491us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:00.162078 (+1094008us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.862217 (+3700139us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54002e68d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:03.862744 (+ 527us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2481692032}
I20250116 03:00:04.033504 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.082259 23936 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:04.026968 20601 rpcz_store.cc:276] 0116 03:00:02.001315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:03.810241 (+1808926us) service_pool.cc:225] Handling call
0116 03:00:03.811634 (+ 1393us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:03.812280 (+ 646us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:03.812373 (+ 93us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:03.812472 (+ 99us) raft_consensus.cc:1640] Marking committed up to 7
0116 03:00:03.812524 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.812625 (+ 101us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:03.812873 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 03:00:04.038293 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.028718 20752 rpcz_store.cc:276] 0116 02:59:59.935218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.997922 (+ 62704us) service_pool.cc:225] Handling call
0116 02:59:59.998331 (+ 409us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:00.166699 (+168368us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:00.166734 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:00.166781 (+ 47us) raft_consensus.cc:1640] Marking committed up to 5
0116 03:00:00.166812 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.887124 (+3720312us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5400261954. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:03.887223 (+ 99us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:03.887651 (+ 428us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":386626432}
I20250116 03:00:04.038321 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 03:00:04.041604 20680 rpcz_store.cc:269] 0116 03:00:01.840897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:01.841080 (+ 183us) service_pool.cc:225] Handling call
0116 03:00:03.892101 (+2051021us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b54001923d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:03.941517 (+ 49416us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4534038784}
I20250116 03:00:04.028498 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.029721 20674 rpcz_store.cc:276] 0116 02:59:59.568100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.568365 (+ 265us) service_pool.cc:225] Handling call
0116 02:59:59.584437 (+ 16072us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:00.248488 (+664051us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.875329 (+3626841us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b5400022bd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:03.875784 (+ 455us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1525454592}
I20250116 03:00:04.051883 20600 raft_consensus.cc:2388] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 4: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
W20250116 03:00:04.024250 20605 rpcz_store.cc:269] 0116 02:59:59.508363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.508541 (+ 178us) service_pool.cc:225] Handling call
0116 03:00:03.860178 (+4351637us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b540024ead4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:03.860425 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5317909248}
I20250116 03:00:04.014456 24005 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:04.029742 21329 rpcz_store.cc:276] 0116 03:00:02.364305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.535758 (+171453us) service_pool.cc:225] Handling call
0116 03:00:02.536245 (+ 487us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:03.587644 (+1051399us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:03.904422 (+316778us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b540078f5d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:03.904869 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1968302848}
I20250116 03:00:04.054659 23580 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:04.056783 20603 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
W20250116 03:00:04.058185 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 127602us
I20250116 03:00:04.058933 20608 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 03:00:04.059063 23812 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250116 03:00:04.059453 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 03:00:04.060741 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 88462.7us
I20250116 03:00:04.068902 20606 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:04.098702 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.099534 20674 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 03:00:04.100621 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:04.102133 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:04.103192 20677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1630) took 2074 ms (client timeout 1955 ms). Trace:
I20250116 03:00:04.097430 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:04.103761 20677 rpcz_store.cc:269] 0116 03:00:02.029156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.410223 (+381067us) service_pool.cc:225] Handling call
0116 03:00:04.102906 (+1692683us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400285d54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:04.103061 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3094151552}
I20250116 03:00:04.104761 20572 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 03:00:04.105453 20677 raft_consensus.cc:1235] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8. Ops: []
I20250116 03:00:04.106024 23580 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 03:00:04.076340 20571 leader_election.cc:336] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 2.113s (SENT)
I20250116 03:00:04.107211 20571 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:04.079218 24170 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 03:00:03.827603 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7658a3a42c44f485085e75a4bd7b88" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:04.034396 23802 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:04.092139 20600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1813) took 2067 ms (client timeout 2132 ms). Trace:
I20250116 03:00:04.091094 20748 raft_consensus.cc:2463] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 03:00:04.092676 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.097563 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 778) took 3952 ms. Trace:
I20250116 03:00:04.092475 24099 consensus_queue.cc:1046] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20250116 03:00:04.095289 20752 raft_consensus.cc:1235] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 6. Ops: []
I20250116 03:00:04.088578 20607 rpcz_store.cc:276] 0116 03:00:02.002647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.452636 (+449989us) service_pool.cc:225] Handling call
0116 03:00:02.452991 (+ 355us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:02.453427 (+ 436us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:02.453459 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 03:00:02.453977 (+ 518us) log.cc:844] Serialized 26 byte log entry
0116 03:00:03.235780 (+781803us) raft_consensus.cc:1640] Marking committed up to 8
0116 03:00:03.971644 (+735864us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 03:00:04.019487 (+ 47843us) log.cc:844] Serialized 14 byte log entry
0116 03:00:04.020049 (+ 562us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:04.020166 (+ 117us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:04.087544 (+ 67378us) raft_consensus.cc:1680] finished
0116 03:00:04.087728 (+ 184us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:04.087919 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":320380,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":59,"raft.queue_time_us":4024,"raft.run_cpu_time_us":413,"raft.run_wall_time_us":11222,"spinlock_wait_cycles":8550272,"thread_start_us":1501943,"threads_started":2,"wal-append.queue_time_us":1014292}
I20250116 03:00:04.095562 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:04.097193 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 69703.3us
I20250116 03:00:04.095173 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.103382 20753 raft_consensus.cc:1235] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
I20250116 03:00:04.090360 20754 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 03:00:04.111716 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.088845 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.110203 20756 rpcz_store.cc:276] 0116 03:00:00.145459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:00.183024 (+ 37565us) service_pool.cc:225] Handling call
0116 03:00:00.183289 (+ 265us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:02.412799 (+2229510us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:04.095738 (+1682939us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b5400261454. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:04.097366 (+ 1628us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4871920000}
W20250116 03:00:04.109679 20600 rpcz_store.cc:269] 0116 03:00:02.024767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:03.849423 (+1824656us) service_pool.cc:225] Handling call
0116 03:00:04.092071 (+242648us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 03:00:04.114284 20680 raft_consensus.cc:1270] T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250116 03:00:04.085492 24131 leader_election.cc:290] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:04.069496 21300 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:04.121491 24107 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250116 03:00:04.122447 23823 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 03:00:04.121634 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:04.124289 20599 raft_consensus.cc:2388] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:04.126709 20599 raft_consensus.cc:1270] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250116 03:00:04.113224 20755 raft_consensus.cc:2463] T 1d7658a3a42c44f485085e75a4bd7b88 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 03:00:04.099701 24031 raft_consensus.cc:3049] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 7 LEADER]: Stepping down as leader of term 7
I20250116 03:00:04.141428 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.138010 24031 raft_consensus.cc:738] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:04.142347 23957 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:04.143471 20599 raft_consensus.cc:1270] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250116 03:00:04.143848 24031 consensus_queue.cc:260] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:04.145716 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.145881 24031 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
W20250116 03:00:04.150681 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 84203.1us
I20250116 03:00:04.155265 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.159358 20602 raft_consensus.cc:1235] T 59601d2a275b4efdbbc89dbcb608871c P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
I20250116 03:00:04.162444 20675 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 03:00:04.163515 24029 raft_consensus.cc:491] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:04.164094 24029 raft_consensus.cc:513] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:04.165174 23802 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:04.151368 24045 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 03:00:04.165515 21333 raft_consensus.cc:1270] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20250116 03:00:04.162472 20602 raft_consensus.cc:1270] T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250116 03:00:04.166239 21333 pending_rounds.cc:77] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Aborting all ops after (but not including) 7
I20250116 03:00:04.166702 21333 pending_rounds.cc:99] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20250116 03:00:04.167183 21333 raft_consensus.cc:2883] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 03:00:04.168236 24089 raft_consensus.cc:491] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:04.168967 24089 raft_consensus.cc:513] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:04.168406 20751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 791) took 2135 ms. Trace:
I20250116 03:00:04.170493 20751 rpcz_store.cc:276] 0116 03:00:02.032977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.192515 (+159538us) service_pool.cc:225] Handling call
0116 03:00:02.195735 (+ 3220us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:02.430536 (+234801us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:02.430572 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:02.430635 (+ 63us) raft_consensus.cc:1640] Marking committed up to 8
0116 03:00:02.430667 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:04.167877 (+1737210us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540007cbd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:04.167943 (+ 66us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:04.168198 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":539414272}
I20250116 03:00:04.168246 21333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 1044) took 5477 ms. Trace:
I20250116 03:00:04.172641 20750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 779) took 4025 ms. Trace:
I20250116 03:00:04.154649 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.174501 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.173645 21333 rpcz_store.cc:276] 0116 02:59:58.690656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:58.692236 (+ 1580us) service_pool.cc:225] Handling call
0116 02:59:58.721878 (+ 29642us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:02.334118 (+3612240us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b1800112140. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d4ad86 00007fc326c9d1de 00007fc326d06367 00007fc326d07b4e 00007fc326d11c89 00007fc326d12556 00007fc326d0eedf 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694
0116 03:00:03.913241 (+1579123us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b58004ffdb0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326c8b1c3 00007fc326ceeaba 00007fc326d063af 00007fc326d07b4e 00007fc326d11c89 00007fc326d12556 00007fc326d0eedf 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694
0116 03:00:04.167906 (+254665us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:04.168167 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1621302400}
I20250116 03:00:04.182173 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.182864 20607 raft_consensus.cc:2388] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 9: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:04.178679 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1824) took 1008 ms. Trace:
I20250116 03:00:04.183372 20600 rpcz_store.cc:276] 0116 03:00:03.170513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:04.173988 (+1003475us) service_pool.cc:225] Handling call
0116 03:00:04.178574 (+ 4586us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 03:00:04.186014 20604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1811) took 2161 ms (client timeout 1972 ms). Trace:
W20250116 03:00:04.186518 20604 rpcz_store.cc:269] 0116 03:00:02.024182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:04.185615 (+2161433us) service_pool.cc:212] Skipping call since client already timed out
0116 03:00:04.185788 (+ 173us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250116 03:00:04.190866 20572 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:04.192288 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:04.193531 20751 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 03:00:04.196682 23964 leader_election.cc:290] T 6af80665fed7453e80094933a2b5bc2d P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:04.198414 20599 raft_consensus.cc:1235] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250116 03:00:04.190904 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 1070) took 1827 ms. Trace:
I20250116 03:00:04.200661 20572 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:04.202083 20572 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:04.203147 24007 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250116 03:00:04.203930 24007 raft_consensus.cc:2798] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 03:00:04.204392 24007 raft_consensus.cc:491] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:04.204780 24007 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:04.207037 20678 raft_consensus.cc:3054] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 3 FOLLOWER]: Advancing to term 4
I20250116 03:00:04.207533 23812 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:04.210268 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:04.213624 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:04.216361 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:04.196555 24058 raft_consensus.cc:3049] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Stepping down as leader of term 6
I20250116 03:00:04.217259 24058 raft_consensus.cc:738] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:04.218390 24058 consensus_queue.cc:260] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:04.211428 20572 leader_election.cc:304] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:04.200124 21336 rpcz_store.cc:276] 0116 03:00:02.363922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.535236 (+171314us) service_pool.cc:225] Handling call
0116 03:00:02.957909 (+422673us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:03.640020 (+682111us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:03.640054 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:03.640095 (+ 41us) raft_consensus.cc:1640] Marking committed up to 6
0116 03:00:03.640128 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:04.190371 (+550243us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b5400421654. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:04.190446 (+ 75us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:04.190701 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1568145152}
I20250116 03:00:04.173082 20750 rpcz_store.cc:276] 0116 03:00:00.147079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:00.203443 (+ 56364us) service_pool.cc:225] Handling call
0116 03:00:00.203817 (+ 374us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:00.653770 (+449953us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:00.653808 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:00.653863 (+ 55us) raft_consensus.cc:1640] Marking committed up to 7
0116 03:00:00.653896 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:04.172148 (+3518252us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b5400370fd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:04.172218 (+ 70us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:04.172426 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1034124160}
I20250116 03:00:04.232697 24099 consensus_queue.cc:1046] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 03:00:04.233021 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:04.235908 20646 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:04.240516 23933 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
W20250116 03:00:04.241643 23760 consensus_meta.cc:287] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.645s user 0.012s sys 0.012s
I20250116 03:00:04.243641 24088 leader_election.cc:290] T e1b6569e42b74436b3ec62c56499eb17 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:04.246055 20649 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 03:00:04.249974 20675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1611) took 5203 ms (client timeout 2227 ms). Trace:
I20250116 03:00:04.253710 24029 leader_election.cc:290] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:04.257315 24089 leader_election.cc:290] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:04.262336 24108 raft_consensus.cc:2798] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 03:00:04.263041 24108 raft_consensus.cc:491] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:04.263505 24108 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:04.274178 24117 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:04.275513 20675 rpcz_store.cc:269] 0116 02:59:59.046059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.048359 (+ 2300us) service_pool.cc:225] Handling call
0116 03:00:04.249684 (+5201325us) spinlock_profiling.cc:243] Waited 5.11 s on lock 0x7b54002a49d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:04.249843 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11750130432}
W20250116 03:00:04.290781 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 55212.4us
I20250116 03:00:04.294093 20608 raft_consensus.cc:2463] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 03:00:04.242158 23760 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:04.302199 20723 leader_election.cc:336] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.536s (SENT)
I20250116 03:00:04.302892 20723 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:04.304028 24119 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:04.280223 24117 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:04.315790 20606 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:04.320467 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1820) took 1151 ms. Trace:
W20250116 03:00:04.300994 21302 leader_election.cc:336] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.512s (SENT)
I20250116 03:00:04.320693 20602 raft_consensus.cc:1235] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 4. Current term is 6. Ops: []
I20250116 03:00:04.320892 20606 rpcz_store.cc:276] 0116 03:00:03.169255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:04.021596 (+852341us) service_pool.cc:225] Handling call
0116 03:00:04.320400 (+298804us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250116 03:00:04.329095 23812 raft_consensus.cc:2743] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:04.329967 23742 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:04.330570 23742 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:04.331904 21328 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 03:00:04.333173 20571 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 03:00:04.340085 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 128878us
I20250116 03:00:04.344450 20572 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:04.346669 20572 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:04.348023 20571 leader_election.cc:304] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:04.333694 21302 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:04.347978 23893 raft_consensus.cc:2743] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 03:00:04.354146 23812 raft_consensus.cc:2798] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 03:00:04.367038 20504 catalog_manager.cc:5526] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "17446c8dacf04461acb843547665250f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } health_report { overall_health: HEALTHY } } }
I20250116 03:00:04.355195 24058 raft_consensus.cc:3054] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:04.375766 20572 leader_election.cc:304] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:04.368381 24131 raft_consensus.cc:2743] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 03:00:04.380376 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 66898.9us
W20250116 03:00:04.386804 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 157859us
I20250116 03:00:04.374295 23812 raft_consensus.cc:491] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 03:00:04.388613 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 93336us
I20250116 03:00:04.386087 23561 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:04.367892 23838 raft_consensus.cc:357] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
W20250116 03:00:04.401202 21302 leader_election.cc:336] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 1.616s (SENT)
I20250116 03:00:04.391069 23812 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:04.386152 24027 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250116 03:00:04.406241 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 58407.8us
I20250116 03:00:04.398711 23838 raft_consensus.cc:738] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Initialized, Role: LEARNER
I20250116 03:00:04.405627 20572 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:04.396540 24122 raft_consensus.cc:513] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:04.417152 24131 raft_consensus.cc:2743] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:04.404932 24027 raft_consensus.cc:491] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 03:00:04.422116 20683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34870 (request call id 1623) took 2395 ms (client timeout 1684 ms). Trace:
I20250116 03:00:04.422025 23561 consensus_queue.cc:1035] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20250116 03:00:04.424789 20683 rpcz_store.cc:269] 0116 03:00:02.027088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.033969 (+ 6881us) service_pool.cc:225] Handling call
0116 03:00:04.421829 (+2387860us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54002fec54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:04.422019 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3739312512}
I20250116 03:00:04.434567 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:04.446805 24091 consensus_queue.cc:1035] T 8229db04eecb4bc3a9ff24fc8bcac934 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20250116 03:00:04.449280 24008 consensus_queue.cc:1035] T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250116 03:00:04.417795 23760 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:04.452096 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:04.452836 24117 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:04.427922 24027 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:04.457149 20749 raft_consensus.cc:2463] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 03:00:04.464603 20678 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 03:00:04.459071 23838 consensus_queue.cc:260] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 03:00:04.467155 23967 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:04.480279 23967 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:04.468680 24051 raft_consensus.cc:2798] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 03:00:04.472045 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:04.438315 24163 raft_consensus.cc:2743] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250116 03:00:04.475423 20675 pending_rounds.cc:77] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: Aborting all ops after (but not including) 2
I20250116 03:00:04.495460 20675 pending_rounds.cc:99] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250116 03:00:04.495820 20675 raft_consensus.cc:2883] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250116 03:00:04.476051 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.496605 20676 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 03:00:04.476871 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.479295 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:04.474900 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.462637 24131 raft_consensus.cc:2798] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 03:00:04.497956 20683 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
W20250116 03:00:04.505368 20678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1846) took 4800 ms (client timeout 2082 ms). Trace:
W20250116 03:00:04.507764 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 46251.4us
W20250116 03:00:04.506950 20678 rpcz_store.cc:269] 0116 02:59:59.704771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:59.704952 (+ 181us) service_pool.cc:225] Handling call
0116 03:00:03.879659 (+4174707us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b180000bac0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d4ad86 00007fc326c9d1de 00007fc326d06367 00007fc326d07b4e 00007fc326d13228 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c
0116 03:00:04.206384 (+326725us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b58007cf4b0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326c8b1c3 00007fc326ceeaba 00007fc326d063af 00007fc326d07b4e 00007fc326d13228 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c
0116 03:00:04.505260 (+298876us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1579865344}
I20250116 03:00:04.520035 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.524708 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:04.529174 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:04.529307 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:04.526826 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.531581 20602 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 8: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:04.529420 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:04.534508 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.535365 20674 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 03:00:04.536623 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.503146 24131 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:04.549190 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:04.550223 20755 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 03:00:04.564392 20756 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 03:00:04.573208 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.574131 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50646 (request call id 1767) took 1996 ms. Trace:
I20250116 03:00:04.523000 23742 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:04.561384 20680 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
I20250116 03:00:04.580231 21329 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:04.542172 24131 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:04.582262 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.586862 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.588136 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.592435 20602 raft_consensus.cc:2388] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 9: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:04.591682 20605 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 03:00:04.596673 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2960f90503b44912b4b4b58cd2ceb89f" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:04.601253 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:04.574524 20757 rpcz_store.cc:276] 0116 03:00:02.421741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:03.252320 (+830579us) service_pool.cc:225] Handling call
0116 03:00:03.265353 (+ 13033us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:03.378958 (+113605us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:03.378996 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 03:00:04.100665 (+721669us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b2c00111c38. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc3256f0741 00007fc3256ffd72 00007fc326d4b141 00007fc326d0f58a 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782
0116 03:00:04.101351 (+ 686us) log.cc:844] Serialized 26 byte log entry
0116 03:00:04.230254 (+128903us) raft_consensus.cc:1640] Marking committed up to 3
0116 03:00:04.246048 (+ 15794us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:04.288508 (+ 42460us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:04.410543 (+122035us) raft_consensus.cc:1680] finished
0116 03:00:04.410566 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:04.418065 (+ 7499us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95219,"raft.queue_time_us":112328,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":66698,"spinlock_wait_cycles":26350976,"thread_start_us":123686,"threads_started":1,"wal-append.queue_time_us":179245}
I20250116 03:00:04.607438 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.606105 20676 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 03:00:04.594838 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:04.618166 21300 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
W20250116 03:00:04.627080 24176 log.cc:926] Time spent T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.201s user 0.001s sys 0.001s
I20250116 03:00:04.630122 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:04.630570 20504 catalog_manager.cc:5526] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 03:00:04.632242 20504 catalog_manager.cc:5526] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 4 to 5, leader changed from f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 5 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 03:00:04.633939 20504 catalog_manager.cc:5526] T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 5 to 6, leader changed from 231d91433a404468a44ea0a107345aeb (127.18.238.130) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 6 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
I20250116 03:00:04.643297 24029 consensus_queue.cc:1046] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 03:00:04.650977 24130 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250116 03:00:04.651296 24029 raft_consensus.cc:3049] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Stepping down as leader of term 5
I20250116 03:00:04.651674 24130 raft_consensus.cc:491] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:04.652120 24130 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:04.651746 24029 raft_consensus.cc:738] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:04.655149 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:04.657896 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.659108 21300 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:04.661163 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:04.664052 23838 ts_tablet_manager.cc:1428] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent starting tablet: real 0.623s user 0.011s sys 0.005s
W20250116 03:00:04.664664 23838 ts_tablet_manager.cc:1460] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Tablet startup took 624ms
I20250116 03:00:04.665726 20649 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
W20250116 03:00:04.667098 20723 leader_election.cc:336] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195): Timed out: RequestConsensusVote RPC to 127.18.238.129:45195 timed out after 2.116s (SENT)
I20250116 03:00:04.634889 24122 leader_election.cc:290] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:04.672374 20723 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:04.682319 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.687363 24160 consensus_queue.cc:1035] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 03:00:04.687597 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:04.689900 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 47609.7us
I20250116 03:00:04.687398 20571 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:04.691936 21299 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:04.693040 24115 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:04.693392 20602 raft_consensus.cc:2463] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 5.
I20250116 03:00:04.694422 24170 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250116 03:00:04.694912 24170 raft_consensus.cc:491] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:04.695287 24170 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Advancing to term 10
I20250116 03:00:04.701606 24115 raft_consensus.cc:2743] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:04.702804 24027 raft_consensus.cc:513] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:04.705278 24027 leader_election.cc:290] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:04.706650 23677 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 03:00:04.707123 23677 raft_consensus.cc:491] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:04.707494 23677 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:04.709188 21336 raft_consensus.cc:1270] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250116 03:00:04.710534 20680 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 03:00:04.713893 20608 raft_consensus.cc:2463] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7.
I20250116 03:00:04.653354 24029 consensus_queue.cc:260] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 03:00:04.719699 23778 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Advancing to term 9
W20250116 03:00:04.757439 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 43626.6us
I20250116 03:00:04.731465 21336 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250116 03:00:04.732973 24087 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
W20250116 03:00:04.733816 24031 consensus_meta.cc:287] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.554s user 0.005s sys 0.006s
I20250116 03:00:04.734325 23967 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 03:00:04.737939 24007 consensus_meta.cc:287] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.533s user 0.008s sys 0.004s
I20250116 03:00:04.665257 24052 consensus_queue.cc:1035] T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
W20250116 03:00:04.739135 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 79154.9us
W20250116 03:00:04.740001 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 55346.9us
W20250116 03:00:04.748526 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 82897.9us
I20250116 03:00:04.729312 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.720499 23823 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:04.767848 21329 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 9.
I20250116 03:00:04.773720 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.738236 24131 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:04.806246 20571 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 03:00:04.814215 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.800485 20748 raft_consensus.cc:3049] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Stepping down as leader of term 5
I20250116 03:00:04.817391 20748 raft_consensus.cc:738] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:04.821053 20608 raft_consensus.cc:2463] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 10.
I20250116 03:00:04.799990 23933 consensus_queue.cc:1035] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
W20250116 03:00:04.831384 24108 consensus_meta.cc:287] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.567s user 0.013s sys 0.002s
I20250116 03:00:04.831696 23763 consensus_queue.cc:1035] T a3ccbf96c6c547e19daa8f5424fa43ee P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250116 03:00:04.832422 24108 raft_consensus.cc:513] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:04.839116 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:04.848728 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:04.820989 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccbf96c6c547e19daa8f5424fa43ee" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:04.819610 24115 raft_consensus.cc:2743] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250116 03:00:04.865456 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 71032.2us
I20250116 03:00:04.856864 21336 raft_consensus.cc:2388] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
I20250116 03:00:04.867738 21335 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
W20250116 03:00:04.873148 24184 log.cc:926] Time spent T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.063s user 0.000s sys 0.002s
I20250116 03:00:04.783032 23823 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:04.777153 24007 raft_consensus.cc:513] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:04.880126 21334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:04.887615 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:04.899654 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.898962 20608 raft_consensus.cc:1235] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8. Ops: []
I20250116 03:00:04.907090 20599 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 03:00:04.904132 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.913727 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.914880 20604 raft_consensus.cc:2463] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 03:00:04.916596 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.921103 20571 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 03:00:04.889644 20720 connection.cc:232] RPC call timeout handler was delayed by -1.6206108929999345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.271s for a response
I20250116 03:00:04.925572 20602 raft_consensus.cc:1235] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [term 4 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 3. Current term is 4. Ops: []
I20250116 03:00:04.918457 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.917511 20606 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 03:00:04.927212 24027 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 03:00:04.897926 23812 raft_consensus.cc:513] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:04.933843 23812 leader_election.cc:290] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:04.935585 24108 leader_election.cc:290] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:04.947847 20751 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 03:00:04.950336 24086 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 03:00:04.958586 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.959647 20603 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 4.
I20250116 03:00:04.962049 20600 raft_consensus.cc:1235] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 7. Current term is 8. Ops: []
W20250116 03:00:04.971706 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 74685.8us
I20250116 03:00:04.987965 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.989101 20606 raft_consensus.cc:2388] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 03:00:04.991725 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.993278 20603 raft_consensus.cc:1235] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 8. Current term is 9. Ops: []
I20250116 03:00:04.962810 20601 raft_consensus.cc:2388] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 5: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:04.998884 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:04.975895 20748 consensus_queue.cc:260] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:04.959174 24058 consensus_meta.cc:287] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.582s user 0.015s sys 0.000s
I20250116 03:00:05.010226 20602 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 8. Current term is 9. Ops: []
I20250116 03:00:04.961093 24057 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:04.921980 24131 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:05.028995 20649 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 03:00:05.009714 24029 raft_consensus.cc:3054] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:05.039746 20748 raft_consensus.cc:3054] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:05.031566 24031 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:05.045956 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.047667 20571 leader_election.cc:304] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 03:00:05.101039 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 58167.6us
W20250116 03:00:05.103786 20720 connection.cc:232] RPC call timeout handler was delayed by -1.3020114979999562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
W20250116 03:00:05.108006 20720 connection.cc:232] RPC call timeout handler was delayed by -1.1778114979998691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1674s for a response
W20250116 03:00:05.108414 20720 connection.cc:232] RPC call timeout handler was delayed by -1.1688114979999682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1684s for a response
I20250116 03:00:05.129098 20571 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:05.137586 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:05.139601 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 110701us
I20250116 03:00:05.146788 24031 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:05.149941 24031 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:05.153215 20720 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:05.124008 24058 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 03:00:05.154851 21332 raft_consensus.cc:1270] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250116 03:00:05.104159 24131 raft_consensus.cc:2743] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:05.160436 21335 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:05.168002 21333 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 7. Current term is 8. Ops: []
I20250116 03:00:05.169176 20649 leader_election.cc:304] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:05.124008 24119 raft_consensus.cc:491] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:05.175987 24119 raft_consensus.cc:513] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:05.177376 24007 leader_election.cc:290] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.178171 24119 leader_election.cc:290] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:05.170478 23812 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250116 03:00:04.665076 23838 ts_tablet_manager.cc:1462] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Trace:
0116 02:59:52.150122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:52.151165 (+ 1043us) service_pool.cc:225] Handling call
0116 02:59:53.531746 (+1380581us) ts_tablet_manager.cc:928] T 59601d2a275b4efdbbc89dbcb608871c P 57fcfcd5ab01438c9a4baf8a61871aa6: Initiating tablet copy from peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
0116 02:59:54.317109 (+785363us) spinlock_profiling.cc:243] Waited 8.58 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc323db881e 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd27da 00007fc323dd2702 00007fc32bc1d671
0116 02:59:55.295889 (+978780us) tablet_copy_client.cc:368] Tablet copy session begun
0116 02:59:55.490010 (+194121us) tablet_metadata.cc:641] Metadata flushed
0116 02:59:55.490336 (+ 326us) tablet_copy_client.cc:497] Wrote new tablet metadata
0116 02:59:56.699663 (+1209327us) tablet_copy_client.cc:418] Wrote new consensus metadata
0116 02:59:56.750978 (+ 51315us) tablet_replica.cc:206] Creating consensus instance
0116 02:59:56.751579 (+ 601us) ts_tablet_manager.cc:983] Replica registered.
0116 02:59:56.751839 (+ 260us) inbound_call.cc:173] Queueing success response
0116 02:59:57.797280 (+1045441us) spinlock_profiling.cc:243] Waited 8.18 ms on lock 0x7b1800335ce0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc323dd67e8 00007fc323dd9cc1 00007fc323dd9c79 00007fc323dd9c09 00007fc323dd9bd1 00007fc323dd8ecd 00007fc32ce13234 00007fc32ce13069 00007fc323db8fe6
0116 02:59:58.815652 (+1018372us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0116 03:00:04.040625 (+5224973us) ts_tablet_manager.cc:1429] Starting tablet replica
0116 03:00:04.074085 (+ 33460us) tablet_replica.cc:260] Starting instrumentation
0116 03:00:04.220902 (+146817us) tablet_replica.cc:279] Starting consensus
Metrics: {"dns_us":17178,"mutex_wait_us":19008,"raft.queue_time_us":84732,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":8732,"spinlock_wait_cycles":38560256,"tablet-copy.queue_time_us":662257,"tablet-download-pool.queue_time_us":54684,"tablet-download-pool.run_cpu_time_us":1567,"tablet-download-pool.run_wall_time_us":218988,"thread_start_us":1566621,"threads_started":2}
I20250116 03:00:05.179610 23823 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 03:00:05.124743 21328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56662 (request call id 647) took 2755 ms (client timeout 2195 ms). Trace:
I20250116 03:00:05.184654 24170 raft_consensus.cc:513] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.154942 24058 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:05.160635 21331 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:36880
I20250116 03:00:05.208626 21331 raft_consensus.cc:604] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 03:00:05.206890 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.160701 24027 consensus_queue.cc:1046] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20250116 03:00:05.213088 24027 consensus_queue.cc:1225] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer 231d91433a404468a44ea0a107345aeb log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250116 03:00:05.213831 24027 raft_consensus.cc:3049] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Stepping down as leader of term 7
I20250116 03:00:05.214190 24027 raft_consensus.cc:738] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 03:00:05.215361 24027 consensus_queue.cc:260] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:05.217083 24027 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:05.195583 24031 raft_consensus.cc:2798] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 03:00:05.220309 24031 raft_consensus.cc:491] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:05.220782 24031 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 4
I20250116 03:00:05.104079 24057 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:05.203092 21328 rpcz_store.cc:269] 0116 03:00:02.352433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:02.362065 (+ 9632us) service_pool.cc:225] Handling call
0116 03:00:05.049649 (+2687584us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b54004df3d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:05.103672 (+ 54023us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4344317696}
I20250116 03:00:05.225267 24057 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.215210 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92adda8583d41508be4e5a8002cac4d" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 03:00:05.166379 24130 consensus_meta.cc:287] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.514s user 0.011s sys 0.004s
I20250116 03:00:05.245735 20606 raft_consensus.cc:2371] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 7. Re-sending same reply.
I20250116 03:00:05.209085 21331 raft_consensus.cc:612] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 03:00:05.227268 24126 raft_consensus.cc:2798] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 03:00:05.193002 23812 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:05.266453 24170 leader_election.cc:290] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:05.267453 24015 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20250116 03:00:05.269740 23812 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:05.209686 24058 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
W20250116 03:00:05.287252 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 134205us
I20250116 03:00:05.258852 24126 raft_consensus.cc:491] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 03:00:05.295200 21299 leader_election.cc:336] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 1.616s (SENT)
I20250116 03:00:05.295962 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.297905 20606 raft_consensus.cc:1270] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250116 03:00:05.136087 24051 raft_consensus.cc:491] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:05.296586 21299 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
W20250116 03:00:05.294953 23677 consensus_meta.cc:287] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.587s user 0.012s sys 0.003s
I20250116 03:00:05.261616 24130 raft_consensus.cc:513] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.298491 24126 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:05.308845 24051 raft_consensus.cc:3054] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
W20250116 03:00:05.354166 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 162019us
I20250116 03:00:05.343044 23561 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250116 03:00:05.358145 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 49970.2us
I20250116 03:00:05.337220 23677 raft_consensus.cc:513] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.365100 23677 leader_election.cc:290] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.366757 23561 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:05.368541 23540 raft_consensus.cc:2743] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 03:00:05.369292 23561 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.370944 21334 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 6: Already voted for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in this term.
W20250116 03:00:05.373883 23736 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.059s sys 0.082s
W20250116 03:00:05.375361 23736 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s user 0.060s sys 0.082s
W20250116 03:00:05.377923 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 86586.8us
I20250116 03:00:05.379408 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:05.381265 20748 raft_consensus.cc:3054] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:05.389671 20504 catalog_manager.cc:5526] T 221670297d85466bac4e5a9bfead9b18 P 17446c8dacf04461acb843547665250f reported cstate change: term changed from 2 to 6, leader changed from 17446c8dacf04461acb843547665250f (127.18.238.131) to 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132). New cstate: current_term: 6 leader_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } } }
W20250116 03:00:05.394734 24087 consensus_meta.cc:287] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.620s user 0.013s sys 0.000s
I20250116 03:00:05.412974 24087 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 03:00:05.416110 20720 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:05.422166 24087 raft_consensus.cc:2798] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 03:00:05.423236 24087 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:05.423552 20571 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:05.424021 24119 consensus_queue.cc:1035] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250116 03:00:05.425973 24108 raft_consensus.cc:2798] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250116 03:00:05.426406 24108 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 03:00:05.426484 23778 consensus_meta.cc:287] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.666s user 0.010s sys 0.002s
I20250116 03:00:05.426831 24108 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 4 FOLLOWER]: Advancing to term 5
I20250116 03:00:05.424340 24122 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250116 03:00:05.423676 24087 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 03:00:05.422847 20720 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:05.433966 20748 raft_consensus.cc:2463] T fe9318d6bdac457facfcaa97dea377f3 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 6.
W20250116 03:00:05.484519 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 353659us
I20250116 03:00:05.430830 23778 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 03:00:05.466454 20571 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:05.493961 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.496906 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.496896 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af80665fed7453e80094933a2b5bc2d" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.506214 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:05.507635 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.501053 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.465636 24122 raft_consensus.cc:2798] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 03:00:05.515549 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:05.516104 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 93432us
I20250116 03:00:05.515640 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:05.517012 20678 raft_consensus.cc:2405] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250116 03:00:05.517369 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.518002 20676 raft_consensus.cc:3054] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:05.504698 20571 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:05.505502 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:05.425997 24031 raft_consensus.cc:513] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:05.524827 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:05.526878 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 417449us
I20250116 03:00:05.527484 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c9bd4e72374377958093c605e5c6ae" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.526046 20682 raft_consensus.cc:1270] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250116 03:00:05.521199 23893 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.026s sys 0.058s
I20250116 03:00:05.532409 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.533172 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.508582 24115 raft_consensus.cc:2758] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250116 03:00:05.539378 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.533905 20600 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
W20250116 03:00:05.532413 23893 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.708s user 0.026s sys 0.058s
W20250116 03:00:05.537714 24079 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.003s sys 0.030s
W20250116 03:00:05.556159 24079 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.700s user 0.004s sys 0.030s
I20250116 03:00:05.556372 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.555771 20675 raft_consensus.cc:2363] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8.
I20250116 03:00:05.538700 20720 leader_election.cc:304] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 03:00:05.516831 24122 raft_consensus.cc:491] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:05.569726 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.572772 20683 raft_consensus.cc:2388] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 8: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:05.555902 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.576320 20674 raft_consensus.cc:2463] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
I20250116 03:00:05.579351 20604 raft_consensus.cc:1235] T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6. Ops: []
I20250116 03:00:05.529311 24130 leader_election.cc:290] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.582957 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.586311 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.587255 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d2558ee3674a5889434a45c66ac3c1" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.594209 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.595288 20600 raft_consensus.cc:2405] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20250116 03:00:05.596843 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 92257.8us
I20250116 03:00:05.598814 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.599054 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.600026 24058 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:05.601667 21299 leader_election.cc:336] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 1.683s (SENT)
I20250116 03:00:05.570626 24122 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:05.604812 23812 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:05.606802 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.607147 24197 consensus_queue.cc:1035] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250116 03:00:05.608067 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.608204 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.608775 20602 raft_consensus.cc:2463] T eb4a7093fd744af792f09b82b0946a8e P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 5.
I20250116 03:00:05.609727 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.609874 24197 consensus_queue.cc:1046] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250116 03:00:05.610952 23778 raft_consensus.cc:3049] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Stepping down as leader of term 7
I20250116 03:00:05.611307 23778 raft_consensus.cc:738] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:05.612257 23778 consensus_queue.cc:260] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.613804 23778 raft_consensus.cc:3054] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:05.614720 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.616075 23677 raft_consensus.cc:491] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:05.616813 23677 raft_consensus.cc:513] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.607649 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.621791 20681 raft_consensus.cc:1235] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 6. Current term is 7. Ops: []
I20250116 03:00:05.623171 20571 leader_election.cc:304] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 03:00:05.624519 20601 raft_consensus.cc:2388] T 79064d60f0d74dc4bb2733272417faf8 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 8: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:05.625658 20680 raft_consensus.cc:2405] T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20250116 03:00:05.580029 23742 raft_consensus.cc:491] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:05.649816 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a569f56f9d2143869b29c78321047d00" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.656041 21334 raft_consensus.cc:1235] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 8. Current term is 9. Ops: []
W20250116 03:00:05.656878 24029 consensus_meta.cc:287] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.613s user 0.006s sys 0.006s
I20250116 03:00:05.628558 24058 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.660441 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.661414 20680 raft_consensus.cc:2388] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:05.598941 24190 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P f65e8d7f76144fab9725ca32a6328c3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 03:00:05.647854 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 85767.4us
I20250116 03:00:05.667935 20676 raft_consensus.cc:2463] T 81fd75864c6947f09d4e5d31a757c4fa P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 6.
W20250116 03:00:05.671185 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 530609us
W20250116 03:00:05.672277 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 247931us
I20250116 03:00:05.559346 24126 raft_consensus.cc:513] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:05.678481 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b6569e42b74436b3ec62c56499eb17" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:05.666530 21299 leader_election.cc:336] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 1.728s (SENT)
I20250116 03:00:05.687619 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.692509 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c881ef7597c74adfbc9e875bfb6e5f7f" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.697798 20649 leader_election.cc:304] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:05.699086 24205 raft_consensus.cc:2798] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250116 03:00:05.699733 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 52407.8us
I20250116 03:00:05.674904 24198 raft_consensus.cc:2798] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250116 03:00:05.715551 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 92521.2us
I20250116 03:00:05.716526 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:05.714108 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 181503us
I20250116 03:00:05.717690 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.666236 24058 consensus_queue.cc:1046] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20250116 03:00:05.721117 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:05.728111 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.728981 20603 raft_consensus.cc:2463] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 8.
I20250116 03:00:05.731904 20681 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:05.726977 20604 raft_consensus.cc:2371] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 17446c8dacf04461acb843547665250f in term 6. Re-sending same reply.
I20250116 03:00:05.696229 24203 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:05.669812 23742 raft_consensus.cc:513] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.735584 20720 leader_election.cc:400] T da8115c09ba24b95a9969edcc6ca4d8c P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6.
I20250116 03:00:05.699748 24205 raft_consensus.cc:491] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:05.694212 23812 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.740558 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.747182 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:05.758010 20756 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:05.747506 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.748356 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:05.749004 20647 leader_election.cc:336] T da8115c09ba24b95a9969edcc6ca4d8c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.530s (SENT)
I20250116 03:00:05.751117 20571 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:05.741868 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:05.744997 21302 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
I20250116 03:00:05.715507 24198 raft_consensus.cc:491] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:05.757298 20600 raft_consensus.cc:2463] T b5ac0297b7614c4a8601eded83fff0a2 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 8.
I20250116 03:00:05.775141 24190 raft_consensus.cc:2743] T 5b1b1a15f38e4ebe9a2943597f827019 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 03:00:05.775751 24193 log.cc:926] Time spent T 54535b75280f444ca16b17d3f05cf9eb P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20250116 03:00:05.777496 24192 log.cc:926] Time spent T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20250116 03:00:05.780117 24031 leader_election.cc:290] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:05.781137 24031 raft_consensus.cc:2798] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 03:00:05.781536 24031 raft_consensus.cc:491] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:05.781908 24031 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:05.784336 23742 leader_election.cc:290] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.785238 24087 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.787132 24126 leader_election.cc:290] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.787820 24108 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:05.795827 24196 raft_consensus.cc:3049] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Stepping down as leader of term 8
I20250116 03:00:05.796335 24196 raft_consensus.cc:738] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:05.796643 24108 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:05.758204 24203 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.797360 24196 consensus_queue.cc:260] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.798954 20606 raft_consensus.cc:2363] T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5.
I20250116 03:00:05.791519 23677 leader_election.cc:290] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:05.769233 24198 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
W20250116 03:00:05.801239 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 52278.4us
W20250116 03:00:05.803661 20647 leader_election.cc:336] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.948s (SENT)
I20250116 03:00:05.804180 20647 leader_election.cc:304] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:05.806478 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.807520 20675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.807783 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229db04eecb4bc3a9ff24fc8bcac934" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.809804 20674 raft_consensus.cc:1270] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 03:00:05.810509 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852e609a81a14d53bed4660b6e8b47f7" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:05.811349 20682 raft_consensus.cc:2371] T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8. Re-sending same reply.
W20250116 03:00:05.817452 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 72533.8us
I20250116 03:00:05.819904 20677 raft_consensus.cc:1235] T 2960f90503b44912b4b4b58cd2ceb89f P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 8. Current term is 9. Ops: []
I20250116 03:00:05.794380 24087 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:05.789834 21300 leader_election.cc:304] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters:
I20250116 03:00:05.771248 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 03:00:05.819746 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 68676.3us
I20250116 03:00:05.828558 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:05.829864 20681 raft_consensus.cc:2388] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:05.827129 20751 raft_consensus.cc:1235] T 54d83a73e060487a8cb3bf72207828c2 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 4. Current term is 5. Ops: []
W20250116 03:00:05.830144 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 111946us
I20250116 03:00:05.834468 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.773569 20723 leader_election.cc:304] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:05.761518 24205 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Advancing to term 5
W20250116 03:00:05.840828 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 51094.6us
I20250116 03:00:05.843704 24089 raft_consensus.cc:2798] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election won for term 6
I20250116 03:00:05.844322 24089 raft_consensus.cc:695] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:05.845286 24089 consensus_queue.cc:237] T fe9318d6bdac457facfcaa97dea377f3 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.740727 24122 raft_consensus.cc:513] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:05.845984 20674 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Advancing to term 10
I20250116 03:00:05.848340 24122 leader_election.cc:290] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:05.832471 20571 leader_election.cc:304] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:05.852502 21302 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:05.764963 20649 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:05.799113 24203 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:05.862721 20756 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 03:00:05.823992 23736 raft_consensus.cc:2743] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 03:00:05.866504 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:05.867183 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.884183 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9318d6bdac457facfcaa97dea377f3" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:05.884445 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 52086.7us
I20250116 03:00:05.871747 20693 tablet_copy_service.cc:342] P 231d91433a404468a44ea0a107345aeb: Request end of tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-59601d2a275b4efdbbc89dbcb608871c received from {username='slave'} at 127.0.0.1:41212
I20250116 03:00:05.872833 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:05.885221 20677 raft_consensus.cc:2388] T fe9318d6bdac457facfcaa97dea377f3 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:05.873102 23677 raft_consensus.cc:2798] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250116 03:00:05.872748 21302 leader_election.cc:304] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:05.827831 24130 raft_consensus.cc:2798] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250116 03:00:05.889331 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 85728.1us
I20250116 03:00:05.885290 20571 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:05.888901 23677 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:05.899605 20693 tablet_copy_service.cc:434] P 231d91433a404468a44ea0a107345aeb: ending tablet copy session 57fcfcd5ab01438c9a4baf8a61871aa6-59601d2a275b4efdbbc89dbcb608871c on tablet 59601d2a275b4efdbbc89dbcb608871c with peer 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:05.899960 23677 raft_consensus.cc:3054] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Advancing to term 11
I20250116 03:00:05.902074 24108 raft_consensus.cc:2743] T 2960f90503b44912b4b4b58cd2ceb89f P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250116 03:00:05.837914 24196 raft_consensus.cc:3054] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:05.904589 20720 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters:
I20250116 03:00:05.919903 20674 raft_consensus.cc:2463] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 10.
I20250116 03:00:05.928625 20571 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb
W20250116 03:00:05.931115 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 92300.7us
W20250116 03:00:05.932293 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 172498us
I20250116 03:00:05.933465 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:05.940471 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 196566us
I20250116 03:00:05.903134 24130 raft_consensus.cc:491] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:05.954214 20756 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 03:00:05.953703 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8115c09ba24b95a9969edcc6ca4d8c" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:05.917883 24203 raft_consensus.cc:2743] T ff78e82fe8da4dd58822fb1d565eb3be P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250116 03:00:05.976109 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 47616.3us
I20250116 03:00:05.977236 20571 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:05.924307 24126 raft_consensus.cc:2743] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250116 03:00:05.925637 24108 raft_consensus.cc:2798] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250116 03:00:05.989779 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 219040us
I20250116 03:00:05.990381 20723 leader_election.cc:304] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:05.989089 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:05.988306 21302 leader_election.cc:304] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f
I20250116 03:00:05.946863 24209 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 03:00:06.007164 20720 leader_election.cc:400] T 3fd4be964f5040df93f3e609dddcc60a P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 5. Current term is 6.
W20250116 03:00:06.009076 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 69136.1us
I20250116 03:00:05.973762 24130 raft_consensus.cc:3054] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:06.020470 20571 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:06.028246 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:06.042171 21299 leader_election.cc:304] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:05.989879 24190 raft_consensus.cc:2798] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader election won for term 9
I20250116 03:00:05.991498 24108 raft_consensus.cc:491] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 03:00:06.021831 24191 log.cc:926] Time spent T 6af80665fed7453e80094933a2b5bc2d P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.141s user 0.000s sys 0.002s
I20250116 03:00:06.011029 24209 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250116 03:00:06.068778 21302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 93818.6us
I20250116 03:00:06.025627 23823 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:06.005049 23893 raft_consensus.cc:2798] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election won for term 8
I20250116 03:00:06.074522 21334 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 9. Current term is 10. Ops: []
I20250116 03:00:06.044027 24203 raft_consensus.cc:2743] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250116 03:00:06.082906 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 70367.7us
W20250116 03:00:06.094848 20647 leader_election.cc:336] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.036s (SENT)
I20250116 03:00:06.095549 20647 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.048141 24115 raft_consensus.cc:2798] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader election won for term 5
I20250116 03:00:06.098074 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461ef3aac67e4aa49204ce07ff551513" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:06.099464 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:06.048219 24108 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:06.057332 24086 raft_consensus.cc:2798] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader election won for term 7
W20250116 03:00:06.102692 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 60644.6us
W20250116 03:00:06.115212 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 175016us
I20250116 03:00:06.114202 20674 raft_consensus.cc:1270] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250116 03:00:06.025490 24198 raft_consensus.cc:513] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:06.123499 21299 leader_election.cc:304] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.073514 24209 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:06.132987 20601 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8. Ops: []
I20250116 03:00:06.126945 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8680f966644b519da3afccd8095ab2" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:06.126541 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef2f3fbb2a04b7fb68c0bd275697181" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:06.134904 20678 raft_consensus.cc:2463] T 7c8680f966644b519da3afccd8095ab2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 4.
I20250116 03:00:06.084431 23742 raft_consensus.cc:2743] T bddd1f65156348fcb619edd31a8971f8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 03:00:06.126891 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:06.132328 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:06.141794 20608 raft_consensus.cc:2388] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 8: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:06.135183 20606 raft_consensus.cc:3054] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Advancing to term 8
W20250116 03:00:06.129566 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 151542us
I20250116 03:00:06.139530 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b626139c64c14e309e332bc7605ba6f1" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:06.139578 20676 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
I20250116 03:00:06.144152 20676 raft_consensus.cc:604] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 03:00:06.144508 20676 raft_consensus.cc:612] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 03:00:06.132478 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38583cae882144f48cec12c72ba697cf" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:06.049036 23967 consensus_queue.cc:1046] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20250116 03:00:06.125895 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd4be964f5040df93f3e609dddcc60a" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:06.102025 23893 raft_consensus.cc:695] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 8 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 03:00:06.149449 20602 raft_consensus.cc:1235] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250116 03:00:06.150427 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:06.153815 20674 raft_consensus.cc:2388] T b799fc79422648ca95fd3d30ab97fe28 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:06.151158 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:06.097604 23947 consensus_queue.cc:1035] T 2d25e9e5e0954623a098bf759e01120b P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.048s
I20250116 03:00:06.122113 24029 raft_consensus.cc:491] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:06.156018 20723 leader_election.cc:304] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:06.161260 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
W20250116 03:00:06.145479 24051 consensus_meta.cc:287] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.789s user 0.009s sys 0.006s
I20250116 03:00:06.165904 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:06.147279 24024 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:06.144774 24058 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250116 03:00:06.087800 23823 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:06.165400 20720 leader_election.cc:304] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.165140 24197 raft_consensus.cc:3049] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Stepping down as leader of term 7
I20250116 03:00:06.197952 20723 leader_election.cc:304] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.191256 24197 raft_consensus.cc:738] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:06.200562 24197 consensus_queue.cc:260] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:06.202049 24197 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:06.204883 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:06.206723 20678 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P 231d91433a404468a44ea0a107345aeb [term 10 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 9. Current term is 10. Ops: []
I20250116 03:00:06.144455 24129 consensus_queue.cc:1046] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.243s
I20250116 03:00:06.205849 20676 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
W20250116 03:00:06.218817 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 53537.4us
I20250116 03:00:06.220880 20720 leader_election.cc:304] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.223984 24087 raft_consensus.cc:2743] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:06.224467 23893 consensus_queue.cc:237] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:06.207389 24162 raft_consensus.cc:2798] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 03:00:06.226557 20723 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:06.226953 23605 raft_consensus.cc:2743] T 8229db04eecb4bc3a9ff24fc8bcac934 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 03:00:06.227358 23812 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 03:00:06.227598 20723 leader_election.cc:304] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:06.228273 24214 raft_consensus.cc:3054] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:06.227941 23812 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:06.228844 23823 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:06.230567 20720 leader_election.cc:304] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.220873 20606 raft_consensus.cc:2463] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
I20250116 03:00:06.196616 24211 raft_consensus.cc:2798] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250116 03:00:06.229102 24212 raft_consensus.cc:2798] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 03:00:06.244566 20676 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:48834
W20250116 03:00:06.233587 20505 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41956 (request call id 336) took 27958 ms (client timeout 15000 ms). Trace:
I20250116 03:00:06.245455 20676 raft_consensus.cc:604] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 03:00:06.245865 20676 raft_consensus.cc:612] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250116 03:00:06.174635 24051 raft_consensus.cc:513] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:06.178138 24029 raft_consensus.cc:513] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:06.259783 21302 leader_election.cc:304] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:06.196929 24160 consensus_queue.cc:1046] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20250116 03:00:06.183427 24024 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:06.277807 20723 leader_election.cc:304] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:06.287952 20677 raft_consensus.cc:2388] T 3fd4be964f5040df93f3e609dddcc60a P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 6: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:06.241748 24211 raft_consensus.cc:491] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:06.240398 24203 raft_consensus.cc:2743] T f929439d9fe44e75b91fde95f7953378 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250116 03:00:06.272514 20720 leader_election.cc:304] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.291683 20675 raft_consensus.cc:2388] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in current term 8: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:06.244565 24212 raft_consensus.cc:491] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:06.271450 24198 leader_election.cc:290] T 8d16d080f4534834ab906841adf87338 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:06.271968 24119 consensus_queue.cc:1035] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20250116 03:00:06.269454 23812 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:06.269951 24122 raft_consensus.cc:2743] T c881ef7597c74adfbc9e875bfb6e5f7f P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 03:00:06.272111 24087 raft_consensus.cc:2798] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader election won for term 6
I20250116 03:00:06.270020 23605 raft_consensus.cc:2798] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader election won for term 8
I20250116 03:00:06.298161 24211 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:06.387223 20677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bddd1f65156348fcb619edd31a8971f8"
dest_uuid: "231d91433a404468a44ea0a107345aeb"
mode: GRACEFUL
new_leader_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
from {username='slave'} at 127.0.0.1:48834
W20250116 03:00:06.390746 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 286928us
I20250116 03:00:06.315443 24212 raft_consensus.cc:3054] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:06.388034 20677 raft_consensus.cc:604] T bddd1f65156348fcb619edd31a8971f8 P 231d91433a404468a44ea0a107345aeb [term 3 LEADER]: Received request to transfer leadership to TS f65e8d7f76144fab9725ca32a6328c3f
W20250116 03:00:06.401396 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 278000us
W20250116 03:00:06.493458 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 398661us
I20250116 03:00:06.403293 24087 raft_consensus.cc:695] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:06.391634 24203 raft_consensus.cc:2798] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 03:00:06.399102 24122 raft_consensus.cc:2743] T e1b6569e42b74436b3ec62c56499eb17 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:06.431279 24115 raft_consensus.cc:695] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [term 5 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 03:00:06.503122 20647 leader_election.cc:336] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.112s (SENT)
I20250116 03:00:06.506420 20647 leader_election.cc:304] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
W20250116 03:00:06.504266 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 400470us
I20250116 03:00:06.433522 24190 raft_consensus.cc:695] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [term 9 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
W20250116 03:00:06.507681 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 229922us
I20250116 03:00:06.511668 20723 leader_election.cc:400] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Vote denied by peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195) with higher term. Message: Invalid argument: T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5.
I20250116 03:00:06.401199 23605 raft_consensus.cc:695] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:06.512216 20723 leader_election.cc:403] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250116 03:00:06.491142 24058 raft_consensus.cc:491] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:06.497493 23561 raft_consensus.cc:491] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:06.515534 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:06.245473 20505 rpcz_store.cc:269] 0116 02:59:38.111097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:59:38.149304 (+ 38207us) service_pool.cc:225] Handling call
0116 03:00:06.031170 (+27881866us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0116 02:59:57.044402 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0116 02:59:57.045321 (+ 919us) write_op.cc:432] Acquiring schema lock in shared mode
0116 02:59:57.045386 (+ 65us) write_op.cc:435] Acquired schema lock
0116 02:59:57.045399 (+ 13us) tablet.cc:659] Decoding operations
0116 02:59:57.113176 (+ 67777us) write_op.cc:620] Acquiring the partition lock for write op
0116 02:59:57.113282 (+ 106us) write_op.cc:641] Partition lock acquired for write op
0116 02:59:57.113320 (+ 38us) tablet.cc:682] Acquiring locks for 14 operations
0116 02:59:57.345724 (+232404us) tablet.cc:698] Row locks acquired
0116 02:59:57.348019 (+ 2295us) write_op.cc:260] PREPARE: finished
0116 02:59:57.366827 (+ 18808us) write_op.cc:270] Start()
0116 02:59:57.401595 (+ 34768us) write_op.cc:276] Timestamp: P: 1736996397353555 usec, L: 0
0116 02:59:57.401621 (+ 26us) op_driver.cc:348] REPLICATION: starting
0116 02:59:57.403503 (+ 1882us) log.cc:844] Serialized 5658 byte log entry
0116 02:59:57.829906 (+426403us) op_driver.cc:464] REPLICATION: finished
0116 02:59:58.121264 (+291358us) write_op.cc:301] APPLY: starting
0116 02:59:58.121405 (+ 141us) tablet.cc:1364] starting BulkCheckPresence
0116 02:59:58.121844 (+ 439us) tablet.cc:1367] finished BulkCheckPresence
0116 02:59:58.121983 (+ 139us) tablet.cc:1369] starting ApplyRowOperation cycle
0116 02:59:58.131028 (+ 9045us) tablet.cc:1380] finished ApplyRowOperation cycle
0116 02:59:58.131402 (+ 374us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0116 02:59:58.131452 (+ 50us) write_op.cc:312] APPLY: finished
0116 02:59:58.132215 (+ 763us) log.cc:844] Serialized 101 byte log entry
0116 02:59:58.132453 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:58.132615 (+ 162us) write_op.cc:454] Released schema lock
0116 02:59:58.132690 (+ 75us) write_op.cc:341] FINISH: Updating metrics
0116 02:59:58.133148 (+ 458us) write_op.cc:489] Releasing partition, row and schema locks
0116 02:59:58.133185 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14772,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":278372,"apply.run_cpu_time_us":5215,"apply.run_wall_time_us":11913,"num_ops":14,"prepare.queue_time_us":269957,"prepare.run_cpu_time_us":39111,"prepare.run_wall_time_us":491896,"raft.queue_time_us":178609,"raft.run_cpu_time_us":62912,"raft.run_wall_time_us":308128,"replication_time_us":414678,"thread_start_us":561094,"threads_started":4,"wal-append.queue_time_us":195264,"wal-append.run_cpu_time_us":21756,"wal-append.run_wall_time_us":1542347}]]}
I20250116 03:00:06.522816 20608 raft_consensus.cc:1235] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 7. Current term is 8. Ops: []
W20250116 03:00:06.525533 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 253105us
I20250116 03:00:06.527460 20720 leader_election.cc:304] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.534883 20605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
W20250116 03:00:06.515812 24220 log.cc:926] Time spent T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.321s user 0.000s sys 0.001s
I20250116 03:00:06.503660 24203 raft_consensus.cc:491] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:06.542965 20683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:06.536001 20605 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 6.
I20250116 03:00:06.550966 20599 raft_consensus.cc:1235] T 65646a05d3974fe8bd8e187dc68a5a05 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 6. Current term is 7. Ops: []
I20250116 03:00:06.562970 20683 raft_consensus.cc:2388] T 8d16d080f4534834ab906841adf87338 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 7: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:06.565629 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d16d080f4534834ab906841adf87338" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 03:00:06.569067 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 77604.2us
I20250116 03:00:06.491084 24079 consensus_queue.cc:1035] T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
W20250116 03:00:06.582733 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 55405.9us
I20250116 03:00:06.592190 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:06.608876 20607 raft_consensus.cc:2388] T b799fc79422648ca95fd3d30ab97fe28 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 231d91433a404468a44ea0a107345aeb in current term 7: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:06.595269 20599 raft_consensus.cc:1270] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Refusing update from remote peer 231d91433a404468a44ea0a107345aeb: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250116 03:00:06.604987 20571 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:06.612166 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fcf38273074747a64f8e0c6d42de37" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:06.562740 24203 raft_consensus.cc:3054] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:06.565054 24029 leader_election.cc:290] T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 03:00:06.645534 20723 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 134016us
I20250116 03:00:06.646935 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:06.663477 20571 leader_election.cc:304] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 03:00:06.664563 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 79087.9us
I20250116 03:00:06.575243 24122 raft_consensus.cc:2743] T a569f56f9d2143869b29c78321047d00 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 03:00:06.592711 23812 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:06.582396 24243 raft_consensus.cc:2743] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250116 03:00:06.535297 23561 raft_consensus.cc:513] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:06.608934 24051 leader_election.cc:290] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:06.531244 24058 raft_consensus.cc:513] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: NON_VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: true } }
I20250116 03:00:06.688789 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:06.661149 24024 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:06.695245 21299 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.697053 20607 raft_consensus.cc:1235] T d92adda8583d41508be4e5a8002cac4d P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 6. Current term is 7. Ops: []
I20250116 03:00:06.663678 24119 raft_consensus.cc:3054] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Advancing to term 5
W20250116 03:00:06.691612 23778 consensus_meta.cc:287] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.072s user 0.011s sys 0.006s
W20250116 03:00:06.695883 23677 consensus_meta.cc:287] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.795s user 0.015s sys 0.000s
W20250116 03:00:06.699868 20647 leader_election.cc:336] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.126s (SENT)
I20250116 03:00:06.702081 20599 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 8.
I20250116 03:00:06.664816 24232 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250116 03:00:06.720681 20723 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:06.719574 20647 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:06.721895 20680 raft_consensus.cc:2463] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 8.
W20250116 03:00:06.696238 24031 consensus_meta.cc:287] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.914s user 0.009s sys 0.006s
W20250116 03:00:06.726922 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 118137us
W20250116 03:00:06.734786 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 87861.2us
W20250116 03:00:06.737128 21300 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.000s sys 0.012s
W20250116 03:00:06.737816 21300 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.570s user 0.000s sys 0.013s
W20250116 03:00:06.738322 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 589148us
I20250116 03:00:06.745220 20603 raft_consensus.cc:1235] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [term 8 LEADER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 7. Current term is 8. Ops: []
I20250116 03:00:06.660662 24190 consensus_queue.cc:237] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:06.669520 23605 consensus_queue.cc:237] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:06.749436 20720 leader_election.cc:400] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Vote denied by peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223) with higher term. Message: Invalid argument: T 852e609a81a14d53bed4660b6e8b47f7 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 7. Current term is 8.
I20250116 03:00:06.733194 24198 raft_consensus.cc:2798] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 03:00:06.662897 24087 consensus_queue.cc:237] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:06.701855 23812 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:06.757107 24227 log.cc:926] Time spent T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.350s user 0.001s sys 0.000s
I20250116 03:00:06.665069 24231 consensus_queue.cc:1035] T 7c8680f966644b519da3afccd8095ab2 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
W20250116 03:00:06.748606 24226 log.cc:926] Time spent T a3ccbf96c6c547e19daa8f5424fa43ee P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.427s user 0.001s sys 0.000s
W20250116 03:00:06.742410 24225 log.cc:926] Time spent T b0d2558ee3674a5889434a45c66ac3c1 P 17446c8dacf04461acb843547665250f: Append to log took a long time: real 0.303s user 0.000s sys 0.001s
W20250116 03:00:06.757251 24222 log.cc:926] Time spent T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.373s user 0.001s sys 0.001s
I20250116 03:00:06.747450 24232 raft_consensus.cc:491] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:06.741111 24245 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250116 03:00:06.773655 20572 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.504s (SENT) [suppressed 86 similar messages]
I20250116 03:00:06.764899 24244 raft_consensus.cc:491] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
W20250116 03:00:06.782855 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 119462us
I20250116 03:00:06.729470 23677 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:06.689586 24115 consensus_queue.cc:237] T 7c32cae59d8843b993750c4fe4e543e0 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:06.769491 24238 log.cc:926] Time spent T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.269s user 0.000s sys 0.001s
I20250116 03:00:06.758641 24198 raft_consensus.cc:491] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 03:00:06.743608 24122 consensus_queue.cc:1046] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20250116 03:00:06.765795 24079 raft_consensus.cc:2743] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250116 03:00:06.736410 24031 raft_consensus.cc:513] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:06.744447 24023 consensus_queue.cc:1046] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20250116 03:00:06.787528 24239 consensus_queue.cc:1046] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20250116 03:00:06.806185 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cb7aba83e046129200b93f32eb3585" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
W20250116 03:00:06.812006 24221 log.cc:926] Time spent T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
I20250116 03:00:06.793787 24245 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:06.780000 24254 raft_consensus.cc:491] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f65e8d7f76144fab9725ca32a6328c3f)
I20250116 03:00:06.749159 24241 consensus_queue.cc:1046] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.272s
W20250116 03:00:06.877195 20572 leader_election.cc:336] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 1.608s (SENT)
I20250116 03:00:06.922122 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f929439d9fe44e75b91fde95f7953378" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
W20250116 03:00:06.812958 24236 log.cc:926] Time spent T 852e609a81a14d53bed4660b6e8b47f7 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
W20250116 03:00:06.908934 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 209082us
I20250116 03:00:06.926954 20682 raft_consensus.cc:2405] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20250116 03:00:06.818575 24027 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.008s sys 0.017s
I20250116 03:00:06.813980 24198 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:06.932348 20572 leader_election.cc:304] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
W20250116 03:00:06.936614 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 219873us
W20250116 03:00:06.937759 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 188000us
W20250116 03:00:06.926363 24214 consensus_meta.cc:287] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.698s user 0.005s sys 0.006s
W20250116 03:00:06.933657 24238 log.cc:926] Time spent T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
W20250116 03:00:06.942559 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 169974us
W20250116 03:00:06.931538 24027 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.780s user 0.009s sys 0.017s
I20250116 03:00:06.913446 24245 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Advancing to term 9
W20250116 03:00:06.935925 24209 consensus_meta.cc:287] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.785s user 0.007s sys 0.004s
I20250116 03:00:06.947276 20720 leader_election.cc:304] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.948856 24214 raft_consensus.cc:2743] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250116 03:00:06.952593 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 75482.9us
I20250116 03:00:06.953492 24240 raft_consensus.cc:2743] T 221670297d85466bac4e5a9bfead9b18 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:06.800590 24232 raft_consensus.cc:513] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:06.950878 21299 leader_election.cc:304] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:06.951309 24231 raft_consensus.cc:491] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:06.804903 24244 raft_consensus.cc:513] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:06.961024 24234 log.cc:926] Time spent T 6af80665fed7453e80094933a2b5bc2d P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.306s user 0.000s sys 0.001s
I20250116 03:00:06.992570 20754 raft_consensus.cc:1235] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 03:00:06.943967 23561 leader_election.cc:290] T f929439d9fe44e75b91fde95f7953378 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:06.949455 24058 leader_election.cc:290] T 59601d2a275b4efdbbc89dbcb608871c P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 03:00:06.960865 24242 log.cc:926] Time spent T 2d25e9e5e0954623a098bf759e01120b P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.300s user 0.000s sys 0.002s
I20250116 03:00:07.004688 21336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 1091) took 1218 ms. Trace:
W20250116 03:00:07.014791 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 64166.6us
I20250116 03:00:07.015901 21299 leader_election.cc:304] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
W20250116 03:00:07.021195 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 74009us
W20250116 03:00:07.003611 24130 consensus_meta.cc:287] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6: Time spent flushing consensus metadata: real 0.956s user 0.013s sys 0.002s
I20250116 03:00:07.024870 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
W20250116 03:00:07.003345 24246 log.cc:926] Time spent T 54535b75280f444ca16b17d3f05cf9eb P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.241s user 0.000s sys 0.001s
W20250116 03:00:06.996536 24211 consensus_meta.cc:287] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.602s user 0.010s sys 0.001s
I20250116 03:00:07.028820 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:07.032119 21331 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:07.011607 21336 rpcz_store.cc:276] 0116 03:00:05.785835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:05.786009 (+ 174us) service_pool.cc:225] Handling call
0116 03:00:05.786441 (+ 432us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:05.834173 (+ 47732us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:05.834214 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 03:00:05.883119 (+ 48905us) log.cc:844] Serialized 26 byte log entry
0116 03:00:06.044459 (+161340us) raft_consensus.cc:1640] Marking committed up to 5
0116 03:00:06.044502 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:06.044575 (+ 73us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:07.004415 (+959840us) raft_consensus.cc:1680] finished
0116 03:00:07.004440 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:07.004616 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40966,"raft.queue_time_us":48169,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":20909,"thread_start_us":96933,"threads_started":1,"wal-append.queue_time_us":388069}
I20250116 03:00:07.036325 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:06.991863 24073 raft_consensus.cc:3049] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Stepping down as leader of term 4
I20250116 03:00:06.986292 24122 raft_consensus.cc:2743] T 27fcf38273074747a64f8e0c6d42de37 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:06.985795 23812 leader_election.cc:290] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.042950 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:07.046268 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 119116us
I20250116 03:00:06.985957 24214 raft_consensus.cc:491] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:07.048097 21299 leader_election.cc:304] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.044803 20720 leader_election.cc:304] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:06.925693 24254 raft_consensus.cc:513] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.037549 21333 raft_consensus.cc:2463] T f69213cf66934f1a9aa19c9defd3a4d8 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 9.
I20250116 03:00:06.962486 24239 consensus_queue.cc:1046] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20250116 03:00:06.978886 24024 consensus_queue.cc:1046] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20250116 03:00:07.022219 24248 raft_consensus.cc:2743] T 0b57de7b372d4101b7f88debce8eea57 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 03:00:07.061872 21329 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:06.958266 24209 raft_consensus.cc:513] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:07.096009 20647 leader_election.cc:336] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.392s (SENT)
I20250116 03:00:07.054690 21299 leader_election.cc:304] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.096608 20647 leader_election.cc:304] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.044250 24073 raft_consensus.cc:738] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
W20250116 03:00:07.071478 24196 consensus_meta.cc:287] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.152s user 0.014s sys 0.000s
W20250116 03:00:07.072826 24108 consensus_meta.cc:287] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f: Time spent flushing consensus metadata: real 0.945s user 0.013s sys 0.000s
I20250116 03:00:07.073323 24244 leader_election.cc:290] T 852e609a81a14d53bed4660b6e8b47f7 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 03:00:07.073997 24205 consensus_meta.cc:287] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 1.183s user 0.013s sys 0.001s
I20250116 03:00:07.076407 24264 raft_consensus.cc:2798] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 FOLLOWER]: Leader election won for term 10
W20250116 03:00:07.127969 24271 log.cc:926] Time spent T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.056s user 0.000s sys 0.008s
I20250116 03:00:07.076645 24265 raft_consensus.cc:491] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:07.130553 24119 raft_consensus.cc:2743] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195) with higher term. Message: Invalid argument: T f7c662e0a6684e369f470f73b4bfcd80 P f65e8d7f76144fab9725ca32a6328c3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f for earlier term 4. Current term is 5.
W20250116 03:00:07.078007 24249 log.cc:926] Time spent T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f: Append to log took a long time: real 0.264s user 0.002s sys 0.000s
I20250116 03:00:07.131775 24073 consensus_queue.cc:260] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.133328 24073 raft_consensus.cc:3054] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 4 FOLLOWER]: Advancing to term 5
I20250116 03:00:07.078933 24204 raft_consensus.cc:3049] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Stepping down as leader of term 6
I20250116 03:00:07.134528 24204 raft_consensus.cc:738] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:07.135469 24204 consensus_queue.cc:260] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.136917 24204 raft_consensus.cc:3054] T d92adda8583d41508be4e5a8002cac4d P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.055610 24270 raft_consensus.cc:491] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:07.131175 24265 raft_consensus.cc:513] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.062778 24023 raft_consensus.cc:2743] T da8115c09ba24b95a9969edcc6ca4d8c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:06.980751 24231 raft_consensus.cc:513] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:07.090418 23677 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 11 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:07.126164 24205 raft_consensus.cc:513] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:07.148718 24205 leader_election.cc:290] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:07.149132 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1859) took 1032 ms. Trace:
I20250116 03:00:07.149626 20674 raft_consensus.cc:2388] T f7c662e0a6684e369f470f73b4bfcd80 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 5: Already voted for candidate 231d91433a404468a44ea0a107345aeb in this term.
I20250116 03:00:07.087229 24122 raft_consensus.cc:2798] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250116 03:00:07.158715 23873 raft_consensus.cc:3049] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 LEADER]: Stepping down as leader of term 8
I20250116 03:00:07.159193 23873 raft_consensus.cc:738] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:07.149600 20604 rpcz_store.cc:276] 0116 03:00:06.117383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:06.117612 (+ 229us) service_pool.cc:225] Handling call
0116 03:00:06.118086 (+ 474us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:06.432411 (+314325us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:06.432446 (+ 35us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 03:00:06.433452 (+ 1006us) log.cc:844] Serialized 26 byte log entry
0116 03:00:06.651870 (+218418us) raft_consensus.cc:1640] Marking committed up to 4
0116 03:00:06.651913 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:06.651988 (+ 75us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:07.045700 (+393712us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x7b440002494c. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc323c19911 00007fc323ce77c5 00007fc323ce81e2 00007fc32699dda1 00007fc32699968a 00007fc32698d4ae 00007fc32698c7f6 00007fc32698c418 00007fc326995df1 00007fc326995da9 00007fc326995d39
0116 03:00:07.141963 (+ 96263us) raft_consensus.cc:1680] finished
0116 03:00:07.141988 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:07.149071 (+ 7083us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":89613,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":11900,"spinlock_wait_cycles":162812288,"thread_start_us":427492,"threads_started":2,"wal-append.queue_time_us":255389}
I20250116 03:00:07.041419 24130 raft_consensus.cc:513] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.122282 24108 raft_consensus.cc:513] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } }
I20250116 03:00:07.177282 24108 leader_election.cc:290] T 568ee33beba043a39928aff5f3ae0b3a P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 03:00:07.180774 24212 consensus_meta.cc:287] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.747s user 0.007s sys 0.003s
I20250116 03:00:07.042318 24211 raft_consensus.cc:513] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.181411 24212 raft_consensus.cc:513] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:07.157135 24270 raft_consensus.cc:513] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.160228 23873 consensus_queue.cc:260] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.183185 24211 leader_election.cc:290] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.184046 23873 raft_consensus.cc:3054] T 2960f90503b44912b4b4b58cd2ceb89f P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:07.185770 24130 leader_election.cc:290] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:07.186429 21334 raft_consensus.cc:1235] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 5. Current term is 6. Ops: []
I20250116 03:00:07.187239 20757 raft_consensus.cc:1235] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 5. Current term is 6. Ops: []
I20250116 03:00:07.122715 24031 leader_election.cc:290] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 03:00:07.188926 24197 consensus_meta.cc:287] T 79064d60f0d74dc4bb2733272417faf8 P 231d91433a404468a44ea0a107345aeb: Time spent flushing consensus metadata: real 0.986s user 0.013s sys 0.001s
I20250116 03:00:07.188465 24245 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.190671 24231 leader_election.cc:290] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:07.157107 24122 raft_consensus.cc:491] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:07.191612 24122 raft_consensus.cc:3054] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.193590 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210b702a7d64ff9970d02aa1f6b4e24" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:07.194309 20604 raft_consensus.cc:3054] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.068445 24214 raft_consensus.cc:513] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
W20250116 03:00:07.200327 21300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 190587us
I20250116 03:00:07.145576 24264 raft_consensus.cc:695] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:07.153432 24023 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:07.144681 24254 leader_election.cc:290] T 0b57de7b372d4101b7f88debce8eea57 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.208050 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:07.196583 24214 leader_election.cc:290] T 2d25e9e5e0954623a098bf759e01120b P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.211776 20571 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:07.159159 24232 leader_election.cc:290] T ff78e82fe8da4dd58822fb1d565eb3be P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:07.217037 20601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:07.215158 20571 leader_election.cc:304] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:07.181280 24209 leader_election.cc:290] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:07.218328 20571 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:07.220010 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d25e9e5e0954623a098bf759e01120b" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:07.230475 21329 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 03:00:07.236569 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:07.216770 24231 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 03:00:07.236335 24108 consensus_queue.cc:1035] T 461ef3aac67e4aa49204ce07ff551513 P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250116 03:00:07.245239 24270 leader_election.cc:290] T a3ccbf96c6c547e19daa8f5424fa43ee P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.246032 24264 consensus_queue.cc:237] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.247941 24265 leader_election.cc:290] T 0ed2f7240f574d40b93ad33069e1a632 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.251349 24245 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.238807 24232 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250116 03:00:07.226971 24240 raft_consensus.cc:2798] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250116 03:00:07.251969 24231 raft_consensus.cc:491] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 17446c8dacf04461acb843547665250f)
I20250116 03:00:07.226701 24239 consensus_queue.cc:1046] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20250116 03:00:07.267580 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
W20250116 03:00:07.242372 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 144598us
W20250116 03:00:07.262799 20720 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 231994us
I20250116 03:00:07.267858 24240 raft_consensus.cc:491] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
I20250116 03:00:07.269233 21299 leader_election.cc:304] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 03:00:07.262835 20752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35540 (request call id 849) took 1160 ms. Trace:
I20250116 03:00:07.272454 20752 rpcz_store.cc:276] 0116 03:00:06.102146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:06.102347 (+ 201us) service_pool.cc:225] Handling call
0116 03:00:06.121942 (+ 19595us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:06.127159 (+ 5217us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:06.127195 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0116 03:00:06.127243 (+ 48us) raft_consensus.cc:1640] Marking committed up to 7
0116 03:00:06.127275 (+ 32us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:07.262516 (+1135241us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x7b5400370fd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:07.262575 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:07.262758 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11450240}
I20250116 03:00:07.268525 24239 raft_consensus.cc:3049] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Stepping down as leader of term 6
I20250116 03:00:07.273845 24239 raft_consensus.cc:738] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:07.275115 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:07.274785 24239 consensus_queue.cc:260] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.277241 24239 raft_consensus.cc:3054] T 31f82fb4c6a24889be1c1e2fe3e7d3e5 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.273087 20720 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.279146 24245 raft_consensus.cc:3054] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:07.266101 24232 raft_consensus.cc:491] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:07.280481 24232 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:07.280903 20720 leader_election.cc:304] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17446c8dacf04461acb843547665250f; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.281852 24270 raft_consensus.cc:2743] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 03:00:07.282438 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43850 (request call id 1858) took 1170 ms. Trace:
I20250116 03:00:07.282887 20602 rpcz_store.cc:276] 0116 03:00:06.112254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:06.121666 (+ 9412us) service_pool.cc:225] Handling call
0116 03:00:06.139567 (+ 17901us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:06.155007 (+ 15440us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:07.282157 (+1127150us) spinlock_profiling.cc:243] Waited 9.62 ms on lock 0x7b54001f0ed4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:07.282371 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22116736}
I20250116 03:00:07.283525 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41212 (request call id 726) took 1475 ms. Trace:
I20250116 03:00:07.284463 24160 raft_consensus.cc:2743] T 3fd4be964f5040df93f3e609dddcc60a P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250116 03:00:07.285746 24210 log.cc:926] Time spent T b0d2558ee3674a5889434a45c66ac3c1 P 57fcfcd5ab01438c9a4baf8a61871aa6: Append to log took a long time: real 1.158s user 0.000s sys 0.006s
I20250116 03:00:07.281827 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 719) took 1512 ms. Trace:
I20250116 03:00:07.267957 24231 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:07.287938 20680 rpcz_store.cc:276] 0116 03:00:05.805832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:05.806048 (+ 216us) service_pool.cc:225] Handling call
0116 03:00:07.281623 (+1475575us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b54002cb854. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:07.281754 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2102819584}
I20250116 03:00:07.283905 20679 rpcz_store.cc:276] 0116 03:00:05.808311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:05.813812 (+ 5501us) service_pool.cc:225] Handling call
0116 03:00:05.814301 (+ 489us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:05.827741 (+ 13440us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:05.827778 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 03:00:05.859500 (+ 31722us) log.cc:844] Serialized 26 byte log entry
0116 03:00:06.056160 (+196660us) raft_consensus.cc:1640] Marking committed up to 8
0116 03:00:06.582963 (+526803us) log.cc:844] Serialized 14 byte log entry
0116 03:00:07.282906 (+699943us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b18002303e0. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc32ca50931 00007fc32ca5d831 00007fc32699a08c 00007fc323db8bfc 00007fc32c9ca7f9 00007fc323dd5a6b 00007fc323dd3e6f 00007fc323dd3305 00007fc326d073d0 00007fc326d19666 00007fc326d2a1d7
0116 03:00:07.283065 (+ 159us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:07.283148 (+ 83us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:07.283200 (+ 52us) raft_consensus.cc:1680] finished
0116 03:00:07.283311 (+ 111us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:07.283439 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50797,"prepare.queue_time_us":105700,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":151308,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":5488,"spinlock_wait_cycles":6442752,"thread_start_us":761474,"threads_started":3,"wal-append.queue_time_us":226967}
I20250116 03:00:07.273371 24212 leader_election.cc:290] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:07.308418 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba18c5d216643cbb5689397ab22da23" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.287199 24240 raft_consensus.cc:3054] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Advancing to term 10
I20250116 03:00:07.316601 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f9e38171174b70b4373b56dbcd90be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
W20250116 03:00:07.317535 24233 log.cc:926] Time spent T b0d2558ee3674a5889434a45c66ac3c1 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.932s user 0.003s sys 0.000s
W20250116 03:00:07.343554 24252 log.cc:926] Time spent T 27fcf38273074747a64f8e0c6d42de37 P 231d91433a404468a44ea0a107345aeb: Append to log took a long time: real 0.612s user 0.004s sys 0.000s
I20250116 03:00:07.344048 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59601d2a275b4efdbbc89dbcb608871c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.344830 20751 raft_consensus.cc:2463] T 59601d2a275b4efdbbc89dbcb608871c P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 03:00:07.345925 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.313782 24198 raft_consensus.cc:513] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:07.319151 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.286702 24270 consensus_queue.cc:1046] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20250116 03:00:07.320430 20602 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8. Ops: []
I20250116 03:00:07.326516 24129 raft_consensus.cc:3049] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Stepping down as leader of term 7
I20250116 03:00:07.327190 20674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34866 (request call id 1908) took 1227 ms. Trace:
I20250116 03:00:07.330261 24266 raft_consensus.cc:3049] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Stepping down as leader of term 6
I20250116 03:00:07.331140 23823 raft_consensus.cc:3049] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 9 LEADER]: Stepping down as leader of term 9
I20250116 03:00:07.332365 23605 consensus_queue.cc:1046] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250116 03:00:07.313122 20752 raft_consensus.cc:2463] T 5ba18c5d216643cbb5689397ab22da23 P 17446c8dacf04461acb843547665250f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 3.
I20250116 03:00:07.346599 20751 raft_consensus.cc:2463] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
W20250116 03:00:07.347302 20675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41212 (request call id 722) took 1540 ms (client timeout 1629 ms). Trace:
W20250116 03:00:07.358270 20675 rpcz_store.cc:269] 0116 03:00:05.807038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:05.807189 (+ 151us) service_pool.cc:225] Handling call
0116 03:00:07.347051 (+1539862us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b54002fdd54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:07.347231 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1104383872}
I20250116 03:00:07.350685 20601 raft_consensus.cc:1235] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 6. Current term is 7. Ops: []
I20250116 03:00:07.349116 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8e1b8edadd42f08808ffaf11b40c8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.351238 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43850 (request call id 1860) took 1204 ms. Trace:
I20250116 03:00:07.361852 20600 rpcz_store.cc:276] 0116 03:00:06.146934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:06.147109 (+ 175us) service_pool.cc:225] Handling call
0116 03:00:07.350997 (+1203888us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002549d4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:07.351148 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2651942144}
I20250116 03:00:07.349447 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.350476 20748 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:07.352191 21334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 1094) took 1062 ms. Trace:
I20250116 03:00:07.352644 20674 rpcz_store.cc:276] 0116 03:00:06.100211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:06.165579 (+ 65368us) service_pool.cc:225] Handling call
0116 03:00:07.326949 (+1161370us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b54002cd154. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d132ae 00007fc32bc6dc37 00007fc32533bc14 00007fc32533bba2 00007fc32533bad1 00007fc32533ba4c 00007fc32533ad02 00007fc32513adfc 00007fc32513a236 00007fc325139bdf
0116 03:00:07.327115 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2254597888}
I20250116 03:00:07.343684 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221670297d85466bac4e5a9bfead9b18" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.366812 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.366211 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81fd75864c6947f09d4e5d31a757c4fa" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.364971 20604 raft_consensus.cc:2463] T f210b702a7d64ff9970d02aa1f6b4e24 P f65e8d7f76144fab9725ca32a6328c3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17446c8dacf04461acb843547665250f in term 7.
I20250116 03:00:07.367720 20752 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 03:00:07.363746 21334 rpcz_store.cc:276] 0116 03:00:06.289734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:06.289942 (+ 208us) service_pool.cc:225] Handling call
0116 03:00:06.315028 (+ 25086us) raft_consensus.cc:1466] Updating replica for 0 ops
0116 03:00:07.186837 (+871809us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:07.351935 (+165098us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b5400520dd4. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:07.352127 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2003545856}
W20250116 03:00:07.334936 24089 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.183s user 0.014s sys 0.001s
W20250116 03:00:07.373808 21299 leader_election.cc:336] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223): Timed out: RequestConsensusVote RPC to 127.18.238.130:45223 timed out after 1.629s (SENT)
I20250116 03:00:07.374315 21299 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.371800 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54535b75280f444ca16b17d3f05cf9eb" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.376869 20723 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17446c8dacf04461acb843547665250f, f65e8d7f76144fab9725ca32a6328c3f; no voters:
W20250116 03:00:07.368468 24086 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.077s user 0.014s sys 0.006s
I20250116 03:00:07.353530 23823 raft_consensus.cc:738] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 03:00:07.352483 24214 consensus_queue.cc:1046] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250116 03:00:07.380769 24262 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 03:00:07.375964 24212 raft_consensus.cc:3049] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Stepping down as leader of term 6
I20250116 03:00:07.335816 24023 raft_consensus.cc:2743] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 03:00:07.386395 20751 raft_consensus.cc:2463] T 54535b75280f444ca16b17d3f05cf9eb P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 03:00:07.352517 24129 raft_consensus.cc:738] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:07.348654 24198 leader_election.cc:290] T 64313b212d37469f98c8d200fb54ca85 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
W20250116 03:00:07.378432 24089 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.222s user 0.014s sys 0.001s
I20250116 03:00:07.353392 24266 raft_consensus.cc:738] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:07.364236 23605 raft_consensus.cc:3049] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Stepping down as leader of term 5
I20250116 03:00:07.394833 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56664 (request call id 1090) took 1772 ms. Trace:
I20250116 03:00:07.395627 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31db17a68ff141f3bd799ccc59c81fcc" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.366138 24162 raft_consensus.cc:491] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 231d91433a404468a44ea0a107345aeb)
W20250116 03:00:07.383792 24086 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.219s user 0.015s sys 0.007s
I20250116 03:00:07.403781 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4a7093fd744af792f09b82b0946a8e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.404249 20755 raft_consensus.cc:2388] T 31db17a68ff141f3bd799ccc59c81fcc P 17446c8dacf04461acb843547665250f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 9: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
I20250116 03:00:07.396040 24245 raft_consensus.cc:2743] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250116 03:00:07.402658 24209 consensus_queue.cc:1046] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250116 03:00:07.404578 20749 raft_consensus.cc:2388] T eb4a7093fd744af792f09b82b0946a8e P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e8d7f76144fab9725ca32a6328c3f in current term 6: Already voted for candidate 17446c8dacf04461acb843547665250f in this term.
W20250116 03:00:07.405437 21299 connection.cc:232] RPC call timeout handler was delayed by -1.1340503239999862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
I20250116 03:00:07.406900 20601 raft_consensus.cc:1270] T 81fd75864c6947f09d4e5d31a757c4fa P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Refusing update from remote peer 17446c8dacf04461acb843547665250f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250116 03:00:07.355509 24130 consensus_queue.cc:1046] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20250116 03:00:07.410820 24269 consensus_queue.cc:1035] T 81fd75864c6947f09d4e5d31a757c4fa P 17446c8dacf04461acb843547665250f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250116 03:00:07.407114 24086 raft_consensus.cc:695] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 7 LEADER]: Becoming Leader. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:07.412333 24086 consensus_queue.cc:237] T d92adda8583d41508be4e5a8002cac4d P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.413324 23823 consensus_queue.cc:260] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.386044 24270 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election won for term 7
I20250116 03:00:07.422466 21336 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0"
dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
mode: GRACEFUL
new_leader_uuid: "17446c8dacf04461acb843547665250f"
from {username='slave'} at 127.0.0.1:36880
I20250116 03:00:07.423362 21336 raft_consensus.cc:604] T e8dd85faff294418bdf7bf8a44f21ea0 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 LEADER]: Received request to transfer leadership to TS 17446c8dacf04461acb843547665250f
I20250116 03:00:07.403152 23605 raft_consensus.cc:738] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 57fcfcd5ab01438c9a4baf8a61871aa6, State: Running, Role: LEADER
I20250116 03:00:07.429759 21334 raft_consensus.cc:1235] T 38583cae882144f48cec12c72ba697cf P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 10 LEADER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 9. Current term is 10. Ops: []
I20250116 03:00:07.422873 24224 consensus_queue.cc:1046] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250116 03:00:07.430384 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c32cae59d8843b993750c4fe4e543e0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:07.434942 20751 raft_consensus.cc:3054] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Advancing to term 5
I20250116 03:00:07.399518 24212 raft_consensus.cc:738] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:07.406744 24162 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:07.437906 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.442704 20756 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 03:00:07.439818 21335 raft_consensus.cc:1235] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 11 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 10. Current term is 11. Ops: []
I20250116 03:00:07.385260 24232 raft_consensus.cc:513] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.444609 20607 raft_consensus.cc:2388] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 17446c8dacf04461acb843547665250f in current term 9: Already voted for candidate f65e8d7f76144fab9725ca32a6328c3f in this term.
I20250116 03:00:07.443264 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:07.403227 21328 rpcz_store.cc:276] 0116 03:00:05.622771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 03:00:05.648758 (+ 25987us) service_pool.cc:225] Handling call
0116 03:00:05.710875 (+ 62117us) raft_consensus.cc:1466] Updating replica for 1 ops
0116 03:00:05.849992 (+139117us) raft_consensus.cc:1529] Early marking committed up to index 0
0116 03:00:05.850033 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0116 03:00:05.903315 (+ 53282us) log.cc:844] Serialized 26 byte log entry
0116 03:00:06.159884 (+256569us) raft_consensus.cc:1640] Marking committed up to 6
0116 03:00:06.169198 (+ 9314us) raft_consensus.cc:1692] Filling consensus response to leader.
0116 03:00:07.366350 (+1197152us) spinlock_profiling.cc:243] Waited 75.4 ms on lock 0x7b540074ac54. stack: 00007fc323da3f63 00007fc323da3e60 00007fc3236f7584 00007fc32d1ebff4 00007fc32d1ebf89 00007fc326c946d1 00007fc326d0ff51 00007fc326d0e99e 00007fc32bc6d2f9 00007fc325339694 00007fc325339622 00007fc325339551 00007fc3253394cc 00007fc325338782 00007fc32513adfc 00007fc32513a236
0116 03:00:07.370480 (+ 4130us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0116 03:00:07.372991 (+ 2511us) raft_consensus.cc:1680] finished
0116 03:00:07.377567 (+ 4576us) raft_consensus.cc:1686] UpdateReplicas() finished
0116 03:00:07.394742 (+ 17175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63080,"raft.queue_time_us":71887,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":17315,"spinlock_wait_cycles":173382400,"thread_start_us":156067,"threads_started":1,"wal-append.queue_time_us":231578}
I20250116 03:00:07.447702 24232 leader_election.cc:290] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.425918 24126 raft_consensus.cc:2743] T 19c9bd4e72374377958093c605e5c6ae P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:07.451933 21299 leader_election.cc:304] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.442834 24229 raft_consensus.cc:491] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:07.457434 24229 raft_consensus.cc:513] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } attrs { replace: true } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.458290 21333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:07.457442 23742 raft_consensus.cc:2758] T 461ef3aac67e4aa49204ce07ff551513 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20250116 03:00:07.459041 21333 raft_consensus.cc:3054] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:07.459494 19386 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250116 03:00:07.461534 19386 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250116 03:00:07.462335 19386 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250116 03:00:07.440399 24270 raft_consensus.cc:695] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [term 7 LEADER]: Becoming Leader. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:07.463526 24270 consensus_queue.cc:237] T f210b702a7d64ff9970d02aa1f6b4e24 P 17446c8dacf04461acb843547665250f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
W20250116 03:00:07.467923 20572 leader_election.cc:336] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.195s (SENT)
I20250116 03:00:07.469872 21333 raft_consensus.cc:2463] T d9815466261546de9d937eeb8cd5cd2e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 9.
I20250116 03:00:07.477934 20755 raft_consensus.cc:1235] T eb87db38c5904a70bb4b4b6d11afa1ca P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
W20250116 03:00:07.478588 24203 consensus_meta.cc:287] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f: Time spent flushing consensus metadata: real 0.788s user 0.007s sys 0.005s
I20250116 03:00:07.479118 24122 raft_consensus.cc:513] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.471544 20571 leader_election.cc:304] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:07.482510 20572 leader_election.cc:304] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e8d7f76144fab9725ca32a6328c3f; no voters: 17446c8dacf04461acb843547665250f, 231d91433a404468a44ea0a107345aeb
I20250116 03:00:07.445086 23823 raft_consensus.cc:3054] T 38583cae882144f48cec12c72ba697cf P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Advancing to term 10
I20250116 03:00:07.513877 20748 raft_consensus.cc:1235] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 6. Current term is 7. Ops: []
I20250116 03:00:07.513720 20749 raft_consensus.cc:1270] T 7c8680f966644b519da3afccd8095ab2 P 17446c8dacf04461acb843547665250f [term 4 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250116 03:00:07.505715 24115 raft_consensus.cc:2798] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader election won for term 9
W20250116 03:00:07.516664 21299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.130:45223, user_credentials={real_user=slave}} blocked reactor thread for 64813.5us
I20250116 03:00:07.484696 24262 raft_consensus.cc:491] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 03:00:07.479161 24203 raft_consensus.cc:513] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.524251 24203 leader_election.cc:290] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 9 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.526947 20754 raft_consensus.cc:1235] T b5ac0297b7614c4a8601eded83fff0a2 P 17446c8dacf04461acb843547665250f [term 9 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 8. Current term is 9. Ops: []
I20250116 03:00:07.527201 24122 leader_election.cc:290] T 19c9bd4e72374377958093c605e5c6ae P 17446c8dacf04461acb843547665250f [CANDIDATE]: Term 7 election: Requested vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.507390 24089 consensus_queue.cc:1046] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
W20250116 03:00:07.532824 20572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 64953us
I20250116 03:00:07.534266 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ac0297b7614c4a8601eded83fff0a2" candidate_uuid: "17446c8dacf04461acb843547665250f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:07.555393 20749 raft_consensus.cc:1235] T 96dc04669716445ab04931358be7efbf P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250116 03:00:07.533093 24031 raft_consensus.cc:3049] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Stepping down as leader of term 6
I20250116 03:00:07.546797 24115 raft_consensus.cc:695] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [term 9 LEADER]: Becoming Leader. State: Replica: f65e8d7f76144fab9725ca32a6328c3f, State: Running, Role: LEADER
I20250116 03:00:07.568928 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79064d60f0d74dc4bb2733272417faf8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:07.564592 24263 raft_consensus.cc:2743] T 31db17a68ff141f3bd799ccc59c81fcc P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250116 03:00:07.577610 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99adad9dddaa4121a293aef6fccd1694" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:07.579602 20755 raft_consensus.cc:3054] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:07.582721 24129 consensus_queue.cc:260] T 99adad9dddaa4121a293aef6fccd1694 P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.547869 24212 consensus_queue.cc:260] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:07.534582 23605 consensus_queue.cc:260] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.590553 23605 raft_consensus.cc:3054] T eb4a7093fd744af792f09b82b0946a8e P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:07.596839 21331 raft_consensus.cc:1235] T b626139c64c14e309e332bc7605ba6f1 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Rejecting Update request from peer 231d91433a404468a44ea0a107345aeb for earlier term 7. Current term is 8. Ops: []
I20250116 03:00:07.565807 24031 raft_consensus.cc:738] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:07.596889 24266 consensus_queue.cc:260] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.599864 24031 consensus_queue.cc:260] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } }
I20250116 03:00:07.600576 24266 raft_consensus.cc:3054] T eb87db38c5904a70bb4b4b6d11afa1ca P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.606050 24212 raft_consensus.cc:3054] T 8d16d080f4534834ab906841adf87338 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.548602 24262 raft_consensus.cc:513] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.566135 24231 raft_consensus.cc:513] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.612802 20604 raft_consensus.cc:1235] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Rejecting Update request from peer 17446c8dacf04461acb843547665250f for earlier term 8. Current term is 9. Ops: []
I20250116 03:00:07.612808 21334 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20250116 03:00:07.613448 24262 leader_election.cc:290] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223)
I20250116 03:00:07.614367 24231 leader_election.cc:290] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Requested vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
W20250116 03:00:07.615953 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 846993us
I20250116 03:00:07.616987 24031 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.619658 24115 consensus_queue.cc:237] T d9815466261546de9d937eeb8cd5cd2e P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.629843 20751 raft_consensus.cc:2463] T 7c32cae59d8843b993750c4fe4e543e0 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 03:00:07.624715 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:07.649091 20755 raft_consensus.cc:2463] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:07.645026 24162 raft_consensus.cc:513] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.653941 24162 leader_election.cc:290] T bf7916e71bc94e1eba4ab8fd75257e48 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 9 election: Requested vote from peers f65e8d7f76144fab9725ca32a6328c3f (127.18.238.129:45195), 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647)
I20250116 03:00:07.655391 20600 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 03:00:07.660532 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:07.663285 21331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3fee7b273345558c661eda0710029c" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6"
I20250116 03:00:07.664017 20754 raft_consensus.cc:1270] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Refusing update from remote peer 57fcfcd5ab01438c9a4baf8a61871aa6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250116 03:00:07.663717 20604 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:07.664475 21331 raft_consensus.cc:3054] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:07.670706 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65646a05d3974fe8bd8e187dc68a5a05" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:07.671627 20757 raft_consensus.cc:3054] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.632828 24204 consensus_queue.cc:1046] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
W20250116 03:00:07.674983 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 226235us
I20250116 03:00:07.633737 24245 raft_consensus.cc:491] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:07.678833 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:07.680802 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1b1a15f38e4ebe9a2943597f827019" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:07.684775 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c04d7691954d8e9278d9ba71f359fe" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:07.691462 20604 raft_consensus.cc:2463] T bf7916e71bc94e1eba4ab8fd75257e48 P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 9.
I20250116 03:00:07.688648 20676 raft_consensus.cc:2463] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fcfcd5ab01438c9a4baf8a61871aa6 in term 9.
I20250116 03:00:07.699604 20751 raft_consensus.cc:1235] T 99adad9dddaa4121a293aef6fccd1694 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Rejecting Update request from peer 57fcfcd5ab01438c9a4baf8a61871aa6 for earlier term 7. Current term is 8. Ops: []
W20250116 03:00:07.697747 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 601722us
W20250116 03:00:07.701056 20647 leader_election.cc:336] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 17446c8dacf04461acb843547665250f (127.18.238.131:43647): Timed out: RequestConsensusVote RPC to 127.18.238.131:43647 timed out after 2.528s (SENT)
I20250116 03:00:07.697878 20749 raft_consensus.cc:2463] T 79064d60f0d74dc4bb2733272417faf8 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:07.701716 20680 raft_consensus.cc:1270] T 461ef3aac67e4aa49204ce07ff551513 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20250116 03:00:07.698027 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.129:45195, user_credentials={real_user=slave}} blocked reactor thread for 72545.1us
I20250116 03:00:07.701454 20647 leader_election.cc:304] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 17446c8dacf04461acb843547665250f, 57fcfcd5ab01438c9a4baf8a61871aa6
I20250116 03:00:07.702538 20649 leader_election.cc:304] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:07.703987 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:07.704563 20754 raft_consensus.cc:3054] T bf7916e71bc94e1eba4ab8fd75257e48 P 17446c8dacf04461acb843547665250f [term 8 FOLLOWER]: Advancing to term 9
I20250116 03:00:07.704387 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f" is_pre_election: true
I20250116 03:00:07.706230 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7916e71bc94e1eba4ab8fd75257e48" candidate_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:07.697587 24256 raft_consensus.cc:491] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 57fcfcd5ab01438c9a4baf8a61871aa6)
I20250116 03:00:07.709167 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69213cf66934f1a9aa19c9defd3a4d8" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:07.709300 24256 raft_consensus.cc:513] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.711920 24256 leader_election.cc:290] T 54d83a73e060487a8cb3bf72207828c2 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.693116 24126 consensus_queue.cc:1046] T 5b1b1a15f38e4ebe9a2943597f827019 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20250116 03:00:07.716506 23736 raft_consensus.cc:3049] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Stepping down as leader of term 7
I20250116 03:00:07.717001 23736 raft_consensus.cc:738] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 231d91433a404468a44ea0a107345aeb, State: Running, Role: LEADER
I20250116 03:00:07.670130 24264 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20250116 03:00:07.720696 24254 raft_consensus.cc:2743] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250116 03:00:07.724042 24229 leader_election.cc:290] T e8dd85faff294418bdf7bf8a44f21ea0 P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 231d91433a404468a44ea0a107345aeb (127.18.238.130:45223), 17446c8dacf04461acb843547665250f (127.18.238.131:43647), 57fcfcd5ab01438c9a4baf8a61871aa6 (127.18.238.132:33959)
I20250116 03:00:07.669541 24253 consensus_queue.cc:1046] T 19c9bd4e72374377958093c605e5c6ae P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
W20250116 03:00:07.725658 20646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.238.132:33959, user_credentials={real_user=slave}} blocked reactor thread for 49677.7us
I20250116 03:00:07.732360 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568ee33beba043a39928aff5f3ae0b3a" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:07.733115 20674 raft_consensus.cc:3054] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Advancing to term 6
I20250116 03:00:07.734063 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:07.735601 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff78e82fe8da4dd58822fb1d565eb3be" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:07.740232 20757 raft_consensus.cc:2463] T 65646a05d3974fe8bd8e187dc68a5a05 P 17446c8dacf04461acb843547665250f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 03:00:07.741170 21331 raft_consensus.cc:2463] T 5d3fee7b273345558c661eda0710029c P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 9.
I20250116 03:00:07.694734 24270 raft_consensus.cc:3049] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 8 LEADER]: Stepping down as leader of term 8
I20250116 03:00:07.682166 24239 consensus_queue.cc:1046] T b5ac0297b7614c4a8601eded83fff0a2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20250116 03:00:07.747538 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b799fc79422648ca95fd3d30ab97fe28" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:07.742733 20571 leader_election.cc:304] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:07.748591 20757 raft_consensus.cc:3054] T b799fc79422648ca95fd3d30ab97fe28 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.752151 20682 raft_consensus.cc:1235] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Rejecting Update request from peer f65e8d7f76144fab9725ca32a6328c3f for earlier term 7. Current term is 8. Ops: []
I20250116 03:00:07.717908 23736 consensus_queue.cc:260] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.685573 24245 raft_consensus.cc:513] T 49f9e38171174b70b4373b56dbcd90be P 17446c8dacf04461acb843547665250f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.693848 24231 consensus_queue.cc:1035] T 03c04d7691954d8e9278d9ba71f359fe P f65e8d7f76144fab9725ca32a6328c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20250116 03:00:07.742395 24261 raft_consensus.cc:2798] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250116 03:00:07.766723 20647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.238.131:43647, user_credentials={real_user=slave}} blocked reactor thread for 65695.4us
I20250116 03:00:07.767246 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dc04669716445ab04931358be7efbf" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:07.778264 20679 raft_consensus.cc:2463] T 96dc04669716445ab04931358be7efbf P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 8.
I20250116 03:00:07.769524 20674 raft_consensus.cc:2463] T 568ee33beba043a39928aff5f3ae0b3a P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 6.
I20250116 03:00:07.782873 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64313b212d37469f98c8d200fb54ca85" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb"
I20250116 03:00:07.785487 20678 raft_consensus.cc:3054] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Advancing to term 7
I20250116 03:00:07.788933 21299 leader_election.cc:304] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.795609 20680 raft_consensus.cc:1270] T 7c32cae59d8843b993750c4fe4e543e0 P 231d91433a404468a44ea0a107345aeb [term 5 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250116 03:00:07.783237 20682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed2f7240f574d40b93ad33069e1a632" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:07.745653 24270 raft_consensus.cc:738] T 5d3fee7b273345558c661eda0710029c P 17446c8dacf04461acb843547665250f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 17446c8dacf04461acb843547665250f, State: Running, Role: LEADER
I20250116 03:00:07.799031 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"
I20250116 03:00:07.805610 20649 leader_election.cc:304] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 231d91433a404468a44ea0a107345aeb; no voters: 57fcfcd5ab01438c9a4baf8a61871aa6, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.811631 20681 raft_consensus.cc:1270] T 03c04d7691954d8e9278d9ba71f359fe P 231d91433a404468a44ea0a107345aeb [term 9 FOLLOWER]: Refusing update from remote peer f65e8d7f76144fab9725ca32a6328c3f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20250116 03:00:07.804780 21299 leader_election.cc:304] T 5b1b1a15f38e4ebe9a2943597f827019 P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.814549 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dd85faff294418bdf7bf8a44f21ea0" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "231d91433a404468a44ea0a107345aeb" is_pre_election: true
I20250116 03:00:07.769145 24251 raft_consensus.cc:2743] T eb87db38c5904a70bb4b4b6d11afa1ca P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250116 03:00:07.823441 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d83a73e060487a8cb3bf72207828c2" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" is_pre_election: true
I20250116 03:00:07.776746 24261 raft_consensus.cc:491] T f210b702a7d64ff9970d02aa1f6b4e24 P 231d91433a404468a44ea0a107345aeb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 03:00:07.794937 24250 consensus_queue.cc:1035] T bf7916e71bc94e1eba4ab8fd75257e48 P 231d91433a404468a44ea0a107345aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20250116 03:00:07.754405 23742 consensus_queue.cc:1035] T 6ef2f3fbb2a04b7fb68c0bd275697181 P 57fcfcd5ab01438c9a4baf8a61871aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17446c8dacf04461acb843547665250f" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20250116 03:00:07.767156 24256 consensus_queue.cc:1046] T 65646a05d3974fe8bd8e187dc68a5a05 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20250116 03:00:07.802009 23736 raft_consensus.cc:3054] T b626139c64c14e309e332bc7605ba6f1 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Advancing to term 8
I20250116 03:00:07.847748 20607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b57de7b372d4101b7f88debce8eea57" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f" is_pre_election: true
I20250116 03:00:07.826162 21299 leader_election.cc:304] T 03c04d7691954d8e9278d9ba71f359fe P 57fcfcd5ab01438c9a4baf8a61871aa6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, 57fcfcd5ab01438c9a4baf8a61871aa6; no voters: f65e8d7f76144fab9725ca32a6328c3f
I20250116 03:00:07.772446 24265 raft_consensus.cc:2798] T 5d3fee7b273345558c661eda0710029c P f65e8d7f76144fab9725ca32a6328c3f [term 9 FOLLOWER]: Leader election won for term 9
I20250116 03:00:07.833994 20748 raft_consensus.cc:2463] T f7c662e0a6684e369f470f73b4bfcd80 P 17446c8dacf04461acb843547665250f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 5.
I20250116 03:00:07.770278 24029 consensus_queue.cc:1046] T 0ed2f7240f574d40b93ad33069e1a632 P 231d91433a404468a44ea0a107345aeb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20250116 03:00:07.839165 20571 leader_election.cc:304] T 96dc04669716445ab04931358be7efbf P f65e8d7f76144fab9725ca32a6328c3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231d91433a404468a44ea0a107345aeb, f65e8d7f76144fab9725ca32a6328c3f; no voters:
I20250116 03:00:07.866232 20678 raft_consensus.cc:2463] T 64313b212d37469f98c8d200fb54ca85 P 231d91433a404468a44ea0a107345aeb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65e8d7f76144fab9725ca32a6328c3f in term 7.
I20250116 03:00:07.804013 24253 raft_consensus.cc:2743] T 0ed2f7240f574d40b93ad33069e1a632 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250116 03:00:07.768992 24240 raft_consensus.cc:513] T f69213cf66934f1a9aa19c9defd3a4d8 P f65e8d7f76144fab9725ca32a6328c3f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f65e8d7f76144fab9725ca32a6328c3f" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 45195 } } peers { permanent_uuid: "231d91433a404468a44ea0a107345aeb" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 45223 } } peers { permanent_uuid: "57fcfcd5ab01438c9a4baf8a61871aa6" member_type: VOTER last_known_addr { host: "127.18.238.132" port: 33959 } attrs { promote: false } }
I20250116 03:00:07.824406 21329 raft_consensus.cc:2463] T 54d83a73e060487a8cb3bf72207828c2 P 57fcfcd5ab01438c9a4baf8a61871aa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231d91433a404468a44ea0a107345aeb in term 5.
I20250116 03:00:07.844252 20603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c662e0a6684e369f470f73b4bfcd80" candidate_uuid: "231d91433a404468a44ea0a107345aeb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f65e8d7f76144fab9725ca32a6328c3f"
I20250116 03:00:07.816816 24015 raft_consensus.cc:2743] T 49cb7aba83e046129200b93f32eb3585 P 231d91433a404468a44ea0a107345aeb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250116 03:00:07.847376 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9815466261546de9d937eeb8cd5cd2e" candidate_uuid: "f65e8d7f76144fab9725ca32a6328c3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "17446c8dacf04461acb843547665250f"